Secrets Error in 2.6 (on Azure)

mabster · December 14, 2021, 9:06pm

I can’t say for sure that hosting on Azure is the problem here, but when I try to set a secret I get an error toast, and the logs have this:

2021-12-14 21:03:50.760 +00:00 [ERR] Failed to set secret :The 'Write-Verbose' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.
2021-12-14 21:03:50.768 +00:00 [ERR] An unhandled exception has occurred while executing the request.
System.Exception: Failed to set variable in secret management.
   at UniversalAutomation.VariableController.Post(Variable variable) in D:\a\universal\universal\src\Universal.Server\Controllers\VariableController.cs:line 161

I’ve set Secrets__SecretStore__Password in the environment variable to a new, random password before deploying. Hopefully that was the right thing to do.

Can paste more of the stack trace if it helps!

Cheers,
Matt

Product: PowerShell Universal
Version: 2.6.0

mabster · December 14, 2021, 9:26pm

Wait - should I be seeing a separate “PSUSecretStore” vault in the dropdown when setting a secret? I still only see “BuiltInLocalVault”.

mabster · December 14, 2021, 9:50pm

OK, I added the “Secrets” section to appsettings.json and redeployed (I still had the old appsettings.json 'coz I’m using the GitHub deployment method) and now I see PSUSecretStore in the vault dropdown.

However, I get the same error/exception when trying to add a secret variable when picking the new vault.

mabster · December 14, 2021, 9:54pm

Oh! This might help! I found this earlier in the log, as the app starts up:

2021-12-14 21:46:33.460 +00:00 [INF] Azure Web Sites environment detected. Using 'D:\home\ASP.NET\DataProtection-Keys' as key repository; keys will not be encrypted at rest.
2021-12-14 21:46:42.980 +00:00 [ERR] Error registering vault PSUSecretStore
2021-12-14 21:46:42.982 +00:00 [ERR] Exception:
System.TypeInitializationException: The type initializer for 'Microsoft.PowerShell.SecretStore.SecureStoreFile' threw an exception.
 ---> System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.
   at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
   at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
   at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
   at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections)
   at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
   at System.IO.FileSystemAclExtensions.SetAccessControl(DirectoryInfo directoryInfo, DirectorySecurity directorySecurity)
   at Microsoft.PowerShell.SecretStore.SecureStoreFile.SetDirectoryACLs(String directoryPath)
   at Microsoft.PowerShell.SecretStore.SecureStoreFile..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.PowerShell.SecretStore.SecureStoreFile.get_ConfigRequiresPassword()
   at Microsoft.PowerShell.SecretStore.LocalSecretStore.get_PasswordRequired()
   at Microsoft.PowerShell.SecretStore.SetSecretStoreConfiguration.EndProcessing()
   at System.Management.Automation.Cmdlet.DoEndProcessing()
   at System.Management.Automation.CommandProcessorBase.Complete()

adam · December 14, 2021, 11:23pm

The PSUSecretStore is the Microsoft SecureStore module. It won’t work in Azure because it’s attempting to write to the user’s local storage location. We implemented support for the new secret management module but we do not yet include direct integration with the Azure Key Vault vault for secret management.

The next step is to get that integrated because I think it’s the right way to work with secrets in Azure. It may be possible to deploy the module and configure it manually.

mabster · December 14, 2021, 11:28pm

OK cool. We’re keen to get into Azure KeyVault so we will pursue that angle!

Do you have any thoughts as to how the KeyVault integration will work? Some sort of setup dialog where you can enter credentials (client id/client secret?) and persist them so secrets are seamlessly available the way they are with the default vault? I might’ve already asked that somewhere else. Just curious.

adam · December 14, 2021, 11:38pm

Yeah we will have to do something like that. The AzKeyVault module takes advantage of the Az module and uses Connect-AzAccount so I would assume that anything we can do with Connect-AzAccount, we can use with the AzKeyVault module.

https://blog.ironmansoftware.com/daily-powershell/powershell-passwords/#store-passwords-in-azure-key-vault

I think from an Azure perspective we’ll need to use Managed Identities. You’ll be able to assign the identity to the web app and then use that identity to connect to the MSI endpoint.

It’s what Azure Functions PowerShell does and that’s also an Azure Web App.

RavenSword · December 15, 2021, 1:26am

I’m getting the same issue with a non-Azure setup, using IIS.

adam · December 15, 2021, 1:49am

mabster:

2021-12-14 21:46:42.982 +00:00 [ERR] Exception:
System.TypeInitializationException: The type initializer for 'Microsoft.PowerShell.SecretStore.SecureStoreFile' threw an exception.
 ---> System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.

Are you seeing this error in the log?

RavenSword · December 15, 2021, 2:02am

Yep on 2.6.
I downgraded to 2.5.5, which is working.

mabster · December 15, 2021, 3:31am

Oh man, we’re so close.

@adam After using the hack you posted in the other thread for installing modules, I have set up our Azure App Service as a system managed identity and given it access to a “PSU” key vault we created.

I can do “Connect-AzAccount -Identity” and “Get-AzKeyVault” without any issues, but “Get-AzKeyVaultSecret” (with the appropriate vault and secret name parameters) is throwing this error:

[error] Cryptography_DpApi_ProfileMayNotBeLoaded

So I’m at the limit of my knowledge now. I’m assuming it’s something to do with PSU not running as a “user” so it can’t do some sort of decryption?

Edit: Interestingly, if I do “-AsPlainText” it does successully read the secret. It’s only when it tries to read it as a securestring with all the metadata that I get the error.

adam · December 15, 2021, 8:55am

@RavenSword - I’ll open an issue for this. It’s a bug.
@mabster - I wonder if you register the Az vault if it will show up in Universal as a vault you can manage from the UI.

https://blog.ironmansoftware.com/daily-powershell/powershell-passwords/#store-passwords-in-azure-key-vault

mabster · December 15, 2021, 10:36am

Where is the best place to register it (and call Connect-AzAccout) so that it persists across all of PSU? If I do it in a “server started” script, will the Azure connection survive?

adam · December 15, 2021, 3:27pm

Connect-AzAccount has a -scope parameter that accepts Process and User. The server started script would be the best but you might need to set it to User if you want to access secrets outside of the PSU process (non-integrated environment).

wakeke · December 15, 2021, 4:16pm

@adam i also see the same problem on 2.6 but i get a different error in the log.

2021-12-15 17:15:09.013 +01:00 [ERR] Failed to set secret :The term 'Write-Verbose' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
2021-12-15 17:15:09.020 +01:00 [ERR] An unhandled exception has occurred while executing the request.
System.Exception: Failed to set variable in secret management.
   at UniversalAutomation.VariableController.Post(Variable variable) in D:\a\universal\universal\src\Universal.Server\Controllers\VariableController.cs:line 147
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 134
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Universal.Server.Middleware.SwaggerAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\universal\universal\src\Universal.Server\Middleware\SwaggerAuthMiddleware.cs:line 40
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
2021-12-15 17:15:09.034 +01:00 [WRN] No exception handler was found, rethrowing original exception.
2021-12-15 17:15:09.034 +01:00 [ERR] Connection id "0HMDVSO5HEF8D", Request id "0HMDVSO5HEF8D:00000006": An unhandled exception was thrown by the application.
System.Exception: Failed to set variable in secret management.
   at UniversalAutomation.VariableController.Post(Variable variable) in D:\a\universal\universal\src\Universal.Server\Controllers\VariableController.cs:line 147
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 134
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Universal.Server.Middleware.SwaggerAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\universal\universal\src\Universal.Server\Middleware\SwaggerAuthMiddleware.cs:line 40
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

adam · December 15, 2021, 4:43pm

The latest nightly build addresses this issue. You may still see the Write-Verbose error but setting secrets will work fine. We’re trying to isolate a crash that some users are experiencing and once that is resolved, we’ll release 2.6.1.

https://imsreleases.z19.web.core.windows.net/

wakeke · December 15, 2021, 4:57pm

I can confirm that this works.

wakeke · December 15, 2021, 5:02pm

@adam I get this info for a newly create PSUSecretStore secret after going into Variables again.
So maybe there is still an issue with the variables?

adam · December 15, 2021, 5:15pm

Can you check the log? I just tried the same thing and it worked…

I’m running PSU as a service with the default account set.

wakeke · December 15, 2021, 6:48pm

Yes it looks like it works now for the BuildInLocalVault but the PSUSecreStore is still showing the same info.

And this is where PSU runs under a gMSA account.

Topic		Replies	Views
Secrets Error in 2.9.1 Universal Automation	0	343	March 10, 2022
Setting Secrets in Azure KeyVault PowerShell Universal	1	265	March 17, 2023
Azure KeyVault Secret Variables PowerShell Universal	11	937	August 11, 2022
Secret variables in Azure keyvault after V5 installation PowerShell Universal	1	30	August 28, 2024
Secrets Vault Missing From "Add Variable" Universal Automation	6	736	July 16, 2020

Secrets Error in 2.6 (on Azure)

Related topics