Calling Connect-AzAccount -Identity -Scope Process in my “On-ServerStarted” script has successfully given me a persistent Azure connection. (I only bothered with process scope because I don’t plan on using anything other than the integrated environment).
Calling Register-SecretVault with my vault name and subscription ID in that same script also made the vault appear in the dropdown when creating a secret variable:
However, when you hit OK to save the variable, you get an error. Here’s the log entry:
System.Exception: Vault AzureKeyVault not found.
at UniversalAutomation.SecretManagerService.UnlockVault(String vaultName, PowerShell powerShell) in D:\a\universal\universal\src\UniversalAutomation.Common\Secrets\SecretManagerService.cs:line 321
at UniversalAutomation.SecretManagerService.SetSecretImpl(String name, Object value, String vault) in D:\a\universal\universal\src\UniversalAutomation.Common\Secrets\SecretManagerService.cs:line 294
2021-12-15 21:00:23.897 +00:00 [ERR] An unhandled exception has occurred while executing the request.
System.Exception: Failed to set variable in secret management.
at UniversalAutomation.VariableController.Post(Variable variable) in D:\a\universal\universal\src\Universal.Server\Controllers\VariableController.cs:line 161
It’s almost like the vault is registering but only for the lifetime of the startup script, and I don’t see a -Scope parameter on Register-SecretVault. Any ideas?
Dec 16, 2021 8:17 AM Hello world!
Dec 16, 2021 8:17 AM [error] Unable to get secret DevWorkshopTest from vault AzureKeyVault
Dec 16, 2021 8:17 AM [error] The secret DevWorkshopTest was not found.
Dec 16, 2021 8:17 AM Name ModuleName IsDefaultVault
Dec 16, 2021 8:17 AM ---- ---------- --------------
Dec 16, 2021 8:17 AM AzureKeyVault Az.KeyVault False
So it can successfully get my “hello world” secret from the vault directly, but not via the SecretManagement module. Very odd!
The issue with the UI is a problem in PSU. We have integrations with SecretStore and the BuiltInLocalVault but since we don’t know what the Az vault is we are throwing an exception. I’ll change this so we can just try to use any vault. I hadn’t thought of that when I suggested registering it. This change will be in 2.6.2.
That’s very weird that you can’t use the secret management vault…I was under the impression it was just calling the Get-AzKeyVaultSecret.
Have done a little more diagnostic digging. I wrote $error[0].Exception from the script when trying to get the secret, and it shows this:
InnerException : System.Management.Automation.RuntimeException: To use PSU
Azure vault, the current user must be logged into Azure
account subscription <Matt's subscription id>.
Run 'Connect-AzAccount -SubscriptionId
<Matt's subscription id>.
But I’ve tried adding the subscription ID to the Connect-AzAccount command and restarting, and the problem persists, so it looks like that’s kind of a “default” exception message. I’ve also tried switching the scope to CurrentUser to no avail.
Running all the same code on my local PC (when connected to Azure as myself) works just fine. I’m baffled.
Worst case scenario I can just use Get-AzKeyVaultSecret in our scripts, but I love the idea of managing the secrets as plain old variables in the UI. Will be great for non-scripty IT users when we get to the point where we can roll this out more broadly.
Y’know, it’s probably all related to that same “DPAPI” error we were getting originally, when you call Get-AzKeyVaultSecret without the -AsPlainText parameter.
Getting closer. I set WEBSITE_LOAD_USER_PROFILE to 1 as an environment variable, and having restarted the app, Get-AzKeyVaultSecret without -AsPlainText succeeds.
However, Get-Secret throws this error:
[error] Vault not found in registry: AzureKeyVault
[warning] There are currently no extension vaults registered.
At least one vault must be registered before SecretManagement can add or retrieve secrets.
You can download SecretManagement extension vault modules from PowerShellGallery.
https://aka.ms/SecretManagementVaults
So it definitely seems like the vault registration in the startup script is not bubbling through to scripts, for some reason.
Ah ha! Tried to re-register the vault from my stand-alone script, and it threw this error:
[error] Unable to find a Local Application Data folder location for the current user, which is needed to store vault registry information.
Windows built-in accounts do not provide the Location Application Data folder and are not currently supported.
So … that’s an issue. Maybe the Az.KeyVault vault needs to be registered by PSU in code somehow, like the built-in and PSU vaults.
Wow I’m being super spammy today. Hope it’s helping in some way and isn’t just an annoyance.
I rolled back the WEBSITE_LOAD_USER_PROFILE setting, because when that was turned on, the SecretManagement module failed entirely. No vaults were loaded at all on system startup.
Interestingly, I was able to create a secret variable using PSUSecretVault just now without an error. Possibly a fix from 2.6.1. I will test a restart of the app and see if it persists.
Edit: Ah, no. Navigated away from the variables page and back to it, and now it’s giving the “secret does not appear in the vault” warning. So no error on creation anymore, but still doesn’t persist, as we expected on Azure.
Creating a secret variable in my “AzureKeyVault” no longer throws an error, and while you’re on the Variables page it looks fine, but if I navigate away from the Variables page and back, I get the same “secret does not appear in the vault” warning next to my secret.
Nothing in the logs other than the SameSite cookie warnings. The new secret definitely doesn’t appear in my vault when I check via the Azure portal.
I’m registering the vault in my “server started” script, like this:
Dang - even if I set the secrets (as PSCredential secrets) manually in the vault, they still don’t show up in PSU. So it’s not just persisting them that’s failing - it’s not even reading their values. It’s like it can’t see the vault at all, even though connecting to Azure and registering the vault seems to be succeeding.
I made a script that simply calls Get-Secret TdCredential -Vault AzureKeyVault (where TdCredential is a secret in my vault).
When I run it, I get these errors:
[error] The 'Write-Verbose' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.
[error] The 'ConvertFrom-Json' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.
[error] Index operation failed; the array index evaluated to null.
[error] Cannot bind argument to parameter 'String' because it is null.
[error] Exception calling ".ctor" with "2" argument(s): "Cannot process argument because the value of argument "userName" is not valid. Change the value of the "userName" argument and run the operation again."
[error] The secret TdCredential was not found.
The ConvertFrom-Json error is a red flag for me, because PSCredential secrets are stored as json in the vault. So maybe PSU can’t find that and it’s why it can’t get secrets out of the vault.
Edit: Interestingly, if I add Import-Module Microsoft.PowerShell.Utility to the top of my script, I get this (yellow) warning:
[warning] The 'Microsoft.PowerShell.Utility' module was not imported because the 'Microsoft.PowerShell.Utility' snap-in was already imported.
So could this be some sort of version mismatch, like we saw with the gallery provider? There’s a version (3.1.0.0) of that module sitting here, I see:
We saw this issue when attempting to fix the crash that others were seeing before. PS7 will automatically add everything from the PSModulePath (even the Windows PowerShell stuff) and then use implicit remoting to create WinPS compatibility sessions and load modules. The weird thing is that it was doing this with Microsoft.PowerShell.Utility. What’s even more strange is that it didn’t do it every time or in every environment. This can obviously cause super weird behavior.
We resolve the crash by synchronizing access to the secret store and then didn’t see this strange behavior.
One thing I was toying with was removing the Windows PowerShell paths from the process-level environment variables manually to prevent this from being possible. The problem is that it changes the behavior of PSU so I would likely need to add an option. That said, I actually built a nightly build that had this option to try in another customer’s environment.
They actually went with the official 2.6.2 release and didn’t need this tweak but if you want to see if it changes your behavior, feel free.
Sorry for all the back and forth on this. The integrate environment can tend to be finicky because all the modules and assemblies are loaded into a single process, and it can be hard to get them to play nice together.
No apologies necessary! It’s frustrating that I can’t just get started building solutions, but I’m happy that I’m helping to improve the Azure-hosting story.
I’ll try doing the update to the nightly build using your command-line guide rather than our GitHub repo. First thing I notice is that on this page:
… the call to Invoke-RestMethod to delete the existing site appears to need a -Method Post parameter. Probably the same for the other commands on that page.
OK I couldn’t get the command-line approach to work so I deployed that nightly build via GitHub instead.
Same behaviour - the secrets show a “does not appear in the vault” warning, and attempting to call Get-Secret from a script throws an error about Microsoft.PowerShell.Utility. When I try to import that module manually, I get this same error:
[warning] The ‘Microsoft.PowerShell.Utility’ module was not imported because the ‘Microsoft.PowerShell.Utility’ snap-in was already imported.
I do notice that you say the nightly build “had this option” - is it a setting I have to change somewhere?
No. It was just enabled by default with no way to turn it off. I’m going to have to follow up with Microsoft about this and change how their module works. It’s causing all kinds of weird problems.
I don’t have a great way to work around this at the moment. I don’t know why you see that error still while I don’t. It’s part of the problem I’m experiencing with this module. It seems to manifest different issues in different environments.
If you don’t have the issue in your Azure instance, then perhaps I just have to make mine look exactly like yours. You’d think it’d be repeatable. Could it be a configuration setting/environment variable or something like that?
Can you post your exact startup script where you override the module path (and presumably do the Connect-AzAccount)? Of course without any PII like subscription ID. That would be a good place to start comparing, I think.
Y’know what’s super weird? I get an error from Get-Secret about how it can’t load Microsoft.PowerShell.Utility to get to ConvertFrom-Json, but the actual ConvertFrom-Json command works just fine in the same script.
[error] The 'Write-Verbose' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.
[error] The 'ConvertFrom-Json' command was found in the module 'Microsoft.PowerShell.Utility', but the module could not be loaded. For more information, run 'Import-Module Microsoft.PowerShell.Utility'.
[error] Index operation failed; the array index evaluated to null.
[error] Cannot bind argument to parameter 'String' because it is null.
[error] Exception calling ".ctor" with "2" argument(s): "Cannot process argument because the value of argument "userName" is not valid. Change the value of the "userName" argument and run the operation again."
[error] The secret TdCredential was not found.
hello
-----
world
The secret module does some very strange stuff. I’m going to work on a PR to patch it in the Microsoft repo. I would have not used it at all if I knew it was going to cause this many problems…
Get-AzKeyVaultSecret -AsPlainText works, so I’m just constructing the credential manually. Once secret vaults are working for us in Azure, I’ll just remove those first three lines and the variable will have a value intrinsically.
Thanks for your work on this, Adam. Looking forward to playing with PSU more in 2022!
That would be a good place to start comparing, I think.