Secrets Error in 2.6 (on Azure)

I can’t say for sure that hosting on Azure is the problem here, but when I try to set a secret I get an error toast, and the logs have this:

2021-12-14 21:03:50.760 +00:00 [ERR] Failed to set secret :The 'Write-Verbose' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.
2021-12-14 21:03:50.768 +00:00 [ERR] An unhandled exception has occurred while executing the request.
System.Exception: Failed to set variable in secret management.
   at UniversalAutomation.VariableController.Post(Variable variable) in D:\a\universal\universal\src\Universal.Server\Controllers\VariableController.cs:line 161

I’ve set Secrets__SecretStore__Password in the environment variable to a new, random password before deploying. Hopefully that was the right thing to do.

Can paste more of the stack trace if it helps!

Cheers,
Matt

Product: PowerShell Universal
Version: 2.6.0

Wait - should I be seeing a separate “PSUSecretStore” vault in the dropdown when setting a secret? I still only see “BuiltInLocalVault”.

OK, I added the “Secrets” section to appsettings.json and redeployed (I still had the old appsettings.json 'coz I’m using the GitHub deployment method) and now I see PSUSecretStore in the vault dropdown.

However, I get the same error/exception when trying to add a secret variable when picking the new vault.

Oh! This might help! I found this earlier in the log, as the app starts up:

2021-12-14 21:46:33.460 +00:00 [INF] Azure Web Sites environment detected. Using 'D:\home\ASP.NET\DataProtection-Keys' as key repository; keys will not be encrypted at rest.
2021-12-14 21:46:42.980 +00:00 [ERR] Error registering vault PSUSecretStore
2021-12-14 21:46:42.982 +00:00 [ERR] Exception:
System.TypeInitializationException: The type initializer for 'Microsoft.PowerShell.SecretStore.SecureStoreFile' threw an exception.
 ---> System.UnauthorizedAccessException: Attempted to perform an unauthorized operation.
   at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
   at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
   at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
   at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections)
   at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
   at System.IO.FileSystemAclExtensions.SetAccessControl(DirectoryInfo directoryInfo, DirectorySecurity directorySecurity)
   at Microsoft.PowerShell.SecretStore.SecureStoreFile.SetDirectoryACLs(String directoryPath)
   at Microsoft.PowerShell.SecretStore.SecureStoreFile..cctor()
   --- End of inner exception stack trace ---
   at Microsoft.PowerShell.SecretStore.SecureStoreFile.get_ConfigRequiresPassword()
   at Microsoft.PowerShell.SecretStore.LocalSecretStore.get_PasswordRequired()
   at Microsoft.PowerShell.SecretStore.SetSecretStoreConfiguration.EndProcessing()
   at System.Management.Automation.Cmdlet.DoEndProcessing()
   at System.Management.Automation.CommandProcessorBase.Complete()

The PSUSecretStore is the Microsoft SecureStore module. It won’t work in Azure because it’s attempting to write to the user’s local storage location. We implemented support for the new secret management module but we do not yet include direct integration with the Azure Key Vault vault for secret management.

The next step is to get that integrated because I think it’s the right way to work with secrets in Azure. It may be possible to deploy the module and configure it manually.

OK cool. We’re keen to get into Azure KeyVault so we will pursue that angle!

Do you have any thoughts as to how the KeyVault integration will work? Some sort of setup dialog where you can enter credentials (client id/client secret?) and persist them so secrets are seamlessly available the way they are with the default vault? I might’ve already asked that somewhere else. Just curious. :slight_smile:

Yeah we will have to do something like that. The AzKeyVault module takes advantage of the Az module and uses Connect-AzAccount so I would assume that anything we can do with Connect-AzAccount, we can use with the AzKeyVault module.

https://blog.ironmansoftware.com/daily-powershell/powershell-passwords/#store-passwords-in-azure-key-vault

I think from an Azure perspective we’ll need to use Managed Identities. You’ll be able to assign the identity to the web app and then use that identity to connect to the MSI endpoint.

It’s what Azure Functions PowerShell does and that’s also an Azure Web App.

1 Like

I’m getting the same issue with a non-Azure setup, using IIS.

Are you seeing this error in the log?

Yep on 2.6.
I downgraded to 2.5.5, which is working.

2 Likes

Oh man, we’re so close.

@adam After using the hack you posted in the other thread for installing modules, I have set up our Azure App Service as a system managed identity and given it access to a “PSU” key vault we created.

I can do “Connect-AzAccount -Identity” and “Get-AzKeyVault” without any issues, but “Get-AzKeyVaultSecret” (with the appropriate vault and secret name parameters) is throwing this error:

[error] Cryptography_DpApi_ProfileMayNotBeLoaded 

So I’m at the limit of my knowledge now. I’m assuming it’s something to do with PSU not running as a “user” so it can’t do some sort of decryption?

Edit: Interestingly, if I do “-AsPlainText” it does successully read the secret. It’s only when it tries to read it as a securestring with all the metadata that I get the error.

1 Like

@RavenSword - I’ll open an issue for this. It’s a bug.
@mabster - I wonder if you register the Az vault if it will show up in Universal as a vault you can manage from the UI.

https://blog.ironmansoftware.com/daily-powershell/powershell-passwords/#store-passwords-in-azure-key-vault

Where is the best place to register it (and call Connect-AzAccout) so that it persists across all of PSU? If I do it in a “server started” script, will the Azure connection survive?

Connect-AzAccount has a -scope parameter that accepts Process and User. The server started script would be the best but you might need to set it to User if you want to access secrets outside of the PSU process (non-integrated environment).

@adam i also see the same problem on 2.6 but i get a different error in the log.

2021-12-15 17:15:09.013 +01:00 [ERR] Failed to set secret :The term 'Write-Verbose' is not recognized as a name of a cmdlet, function, script file, or executable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
2021-12-15 17:15:09.020 +01:00 [ERR] An unhandled exception has occurred while executing the request.
System.Exception: Failed to set variable in secret management.
   at UniversalAutomation.VariableController.Post(Variable variable) in D:\a\universal\universal\src\Universal.Server\Controllers\VariableController.cs:line 147
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 134
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Universal.Server.Middleware.SwaggerAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\universal\universal\src\Universal.Server\Middleware\SwaggerAuthMiddleware.cs:line 40
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
2021-12-15 17:15:09.034 +01:00 [WRN] No exception handler was found, rethrowing original exception.
2021-12-15 17:15:09.034 +01:00 [ERR] Connection id "0HMDVSO5HEF8D", Request id "0HMDVSO5HEF8D:00000006": An unhandled exception was thrown by the application.
System.Exception: Failed to set variable in secret management.
   at UniversalAutomation.VariableController.Post(Variable variable) in D:\a\universal\universal\src\Universal.Server\Controllers\VariableController.cs:line 147
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
   at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
   at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 134
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at Universal.Server.Middleware.SwaggerAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\universal\universal\src\Universal.Server\Middleware\SwaggerAuthMiddleware.cs:line 40
   at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext()
--- End of stack trace from previous location ---
   at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

The latest nightly build addresses this issue. You may still see the Write-Verbose error but setting secrets will work fine. We’re trying to isolate a crash that some users are experiencing and once that is resolved, we’ll release 2.6.1.

https://imsreleases.z19.web.core.windows.net/

3 Likes

I can confirm that this works.

@adam I get this info for a newly create PSUSecretStore secret after going into Variables again.
So maybe there is still an issue with the variables?

image

Can you check the log? I just tried the same thing and it worked… :woozy_face:

I’m running PSU as a service with the default account set.

Yes it looks like it works now for the BuildInLocalVault but the PSUSecreStore is still showing the same info.

And this is where PSU runs under a gMSA account.