Problems using gMSA account for PSU service - Cert error

[Mentat]

Product: PowerShell Universal
Version: 3.1.4

Hey PSU folks,

I’ve been running PSU for about a year now. Installing from MSI and running under a service account without any issues. My Security Team would like me to use gMSA account for running PSU vs a service account. After browsing the forums and seeing people running the service as a gMSA account, I’m trying to get that in place. I’ve created the gMSA, installed on the PSU server, and tested it can be used from the server.

At this point I set the PowerShellUniversal service to run as the gMSA account, and it accepts the account; this grants it the Log on as a Service right. I’ve also added the gMSA account to the Local Administrators group. However, restarting the service (and rebooting) does not work. The service keeps failing with an error related to the cert.

Description: The process was terminated due to an unhandled exception.
Exception Info: System.InvalidOperationException: The requested certificate *.domain.com could not be found in LocalMachine/My with AllowInvalid setting: False.

I’ve tried adding the gMSA account Full Access to the cert’s Private Key. I’ve added additional Local User Rights of Adjust memory quotas for a process (Admins group already there), and Replace a process level token. However, restarting and rebooting still gives the same error.

I’ve rolled back the gMSA account so the service is using the service account, restarted the service, and everything works again.

To note, this is my Kestrel settings in appsettings.json. And as mentioned, everything with the cert works with a service account, just not the gMSA account.

  "Kestrel": {
    "Endpoints": {
      "HTTP": {
        "Url": "http://*:80"
      },
      "HTTPS": {
        "Url": "https://*:443",
        "Certificate": {
          "Subject": "*.<domain>.com",
          "Store": "My",
          "Location": "LocalMachine"
        }
      }
    },
	"RedirectToHttps": "true"
  },

Any ideas of what I need to do so my gMSA account can find and read the cert, so the service will start?

Thanks,
Rob

[coreylista]

I sort of ran into a problem like this recently but it was using MS enterprise CA… this article helped in being able to troubleshoot further to see if the cert store is accessible to the gMSA account.

AD FS – Accessing the certificate store of a gMSA account – Ulysses Neves

[jweigand]

My last team found a workaround for importing certs for the gMSA user - create a scheduled task that ran PowerShell to import the cert, run the scheduled task as the gMSA user. We had a script that made the task, ran it a few seconds later, confirmed it ran successfully, then removed the scheduled task. Bit hacky, but only needed to be done one time per server, and it resolved all our issues with gMSA users being able to see certs they needed.

[Mentat]

Hey y’all,

@jweigand This makes sense, and I’ve gone about it this way. After getting the script to create the scheduled task as the gMSA account; it successfully runs, and triggers the script that installs the certificate as the gMSA account. To prove it I had the script write a file with $env:USERNAME and it’s the gMSA account.

Unfortunately, I still have the same problem and the same error “The requested certificate *.domain.com could not be found in LocalMachine/My”.

I then used the info from @coreylista and was able to launch a PoSh session as the gMSA account. From there I am able to successfully cd Cert:\LocalMachine\My, and see the cert in question; as the gMSA account.

Sigh…It really shouldn’t be this difficult!

Other ideas?

Thanks,
Rob

[Mentat]

To add…I installed the cert under Cert:\CurrentUser\My as the gMSA, and verified the cert exists and can be seen by gMSA.

Updated appsettings.json to point to CurrentUser/My.

Still cannot start PSU service, and still get the same error about cannot find the cert. Of course this time is says it can’t find it in CurrentUser/My.

[adam]

I made a script to help diagnosis problems like this. It replicates the certificate lookup done by ASP.NET Core (and PSU).

HTTPS certificate not found

I’ve done some digging into this. I actually recreated the CertificateLoader class used by ASP.NET in PowerShell. This is the class that is internally being used to look up the cert. Here’s that script. $allowInvalid = $true $subject = 'localhost' $store = [System.Security.Cryptography.X509Certificates.X509Store]::new('My', 'LocalMachine') $store.Open('ReadOnly') $storeCertificates = $store.Certificates; $foundCertificates = $storeCertificates.Find('FindBySubjectName', $Subject, $allowInvalid…

It’s a bit different than the Cert drive etc so it might be worth trying under your gMSA account.

[Mentat]

Hey @adam,

I’ve run that script as my account and it returns the cert. When I run it as the gMSA, it does not return.

When gMSA I see the certs in $storeCertificates, but the next step of $foundCertificates does not show the cert.

Apologies, but I suck at managing certs. What does this mean, and is there a way to fix it so gMSA sees the cert? I’ve already created the cert as gMSA which automatically added gMSA Full Access to the Private Key of the cert.

Thanks,
Rob

[Mentat]

I was playing around, and found if I change the $foundCertificates line to !$allowInvalid, the gMSA does return the cert.

$foundCertificates = $storeCertificates.Find(‘FindBySubjectName’, $Subject, !$allowInvalid);

[Mentat]

Based on this, I added “AllowInvalid”: “true” to my Kesterel settings. It worked…kind of. The PowerShellUniversal service started and I can reach UD pages. However, every page I visit has a UDToast message of “The remote certificate is invalid because of errors in the certificate chain: PartialChain at line 42”, or other line number.

The cert is from GoDaddy, and is valid. When running as service account with AllowInvalid = false, everything works without issue and without cert probs. When running as gMSA with AllowInvalid = true, I can start the service but get the error about PartialChain; this does not make sense to me as we know the cert is valid and works when running as service account and the cert shows valid in the browser.

Thanks,
Rob

[Mentat]

And another update…

I found that UD pages without any calls to PSU APIs work without the PartialChain cert UDToast message. All UD pages that contain PSU API calls, which is almost all of them, give the PartialChain cert UDToast error message.

Anywhere I use Invoke-RestMethod and call a PSU API endpoint script, I have to add -SkipCertificateCheck so I don’t get that PartialChain cert Toast error message. This doesn’t seem right.

So, running as gMSA with AllowInvalid = false will allow the service to start. However, PSU API calls give an error that I have to bypass with -SkipCertificateCheck. The overall issue seems to be that gMSA sees the certificate as invalid, or possibly even as a self-signed cert. This doesn’t make sense to me though. A service account does see it as valid, the cert chain is valid, the cert is working on other servers, and the cert/chain shows valid when viewing through the browser.

Definitely seem to be getting closer. Any ideas from here? I shouldn’t need to use AllowInvalid = True, and shouldn’t need to use -SkipCertificateCheck.

Thanks,
Rob

[adam]

Is it possible that you need to provide some sort of access to the next cert in the chain for the gMSA account? It seems like it is now validating the certificate but failing to validate the root of that certificate.

[Mentat]

Continuing this journey …

@adam Unfortunately I’m not finding anything on assigning permissions to certs in Trusted Root Certification Authorities. But you may be on to something.

I installed the gMSA and the cert on a workstation. Ran both the Test-Certificate cmd and used your script to test the cert for PSU. All tests pass, and they pass for both my user and the gMSA. Heck, on the workstation I installed the cert as my user account and did not bother adding the gMSA to the cert’s private key, and it all passed testing. Something different between the workstation and the server is preventing the gMSA from validating the chain.

I though maybe we’re blocking outbound traffic on the server, preventing the cert validation, but network folks not seeing anything being blocked.

I had the thought to run certlm.msc as the gMSA account and found something interesting. When running as my account, the cert properties look good. The General tab says “This certificate is intended for the following purpose(s):” and lists out some things. The Certification Path tab shows 2 identical Go Daddy Root Certificate Authority - G2 certs, and they show as This certificate is OK.

However, when running as gMSA the General tab says “Windows does not have enough information to verify this certificate”. The Certification Path tab just shows the wild card cert and says “The issuer of this certificate could not be found”. In addition, gMSA can browse to the Go Daddy Root Certificate Authority - G2 cert and view it’s properties; where everything looks good.

Hope this helps shed some light. I feel like we’re getting close, but still lost at this point.

Thanks,
Rob

[Mentat]

Hey everyone,

Our story concludes! Working with one of my server guys last night, we found the Go Daddy intermediate cert was missing from the server. After downloading/installing it, things just magically started working

I reverted things back to the original state; where I’ve installed the cert as my account, no special permissions assigned to the Private Key, Kestrel is set back to AllowInvalid = False, and the PSU service running as a service account. All I had to do is change the PSU service to run as the gMSA account and restart the service. Everything is working just as I originally envisioned, and is actually a very simple setup. Stoopid cert issues…GRRrrrrrr…LOL

Thank you @adam @jweigand @coreylista for the help. I ended up learning a lot from this trial by fire, and your responses definitely helped.

Have a good one!
Rob

[ecosta442]

Hey Rob,

Great news! I just saw this post today, and it caught my eye as I too have started using a gMSA as the service runas account! the only gotcha i had was just making sure the servers you have it installed on have permissions to retrieve and reset the password of the account, and everything works well!

• Evan

[Mentat]

Hey everyone,

Using my lessons learned from doing this in my Dev environment, I’ve just completed a totally smooth Prod upgrade to use gMSA

Figured I’d outline what I did for this, in case it will help others.

I already had my gMSA created and installed on my PSU server. There are plenty of guides on how to do this, but basically:

• Create an AD Security Group, and join your PSU servers as members. Member computers of this group are allowed to use the gMSA.
• Create gMSA, assigning the group to it.
• Add gMSA to any AD groups it needs to be a member of for access permissions.
• Install gMSA on server, I think I rebooted, and run test command.

1. Identify all scripts that use service accounts you’ll be replacing with the gMSA.

2. Installed the intermediate cert I learned about previously.

3. Running PoSh as gMSA, I ran the Test-Certificate command. Success.

4. Join gMSA to Local Administrators group, on PSU server.

5. Grant the following roles to gMSA, on PSU server.

   a. Log on as a batch job
   b. Log on as a service
   c. Adjust memory quotas for a process
   d. Replace a process level token

6. Set the PowerShellUniversal service to run as gMSA.

7. Restart service, give it a couple of mins and review the App Event Log for any errors. Should be none.

8. Test access to the PSU Admin web page.

9. Reboot so the gMSA is running the service with full permissions.

10. Review App Event Log, and test access to the PSU Admin web page.

11. Reset/Recreate secret variables, so gMSA can access them.
    a. Since we’re replacing service accounts with gMSA, I had left over secrets we will no longer use and deleted them.

12. Update all scripts that use service accounts you’re replacing with gMSA.
    a. Remove the importing of service account secret variables.
    b. Remove usage of the -Credential parameter from all commands.
    c. Things will now run as the gMSA, running the service.

13. Test all API Endpoints.

14. Test all Automation Scripts.

15. Test all UD dashboard and page scripts.

16. Test, test, test anything else you may have running in PSU.

Have a good one!
Rob

[sgaglione_laz]

Hey @Mentat,

Can you explain why you added the gMSA account to the local admin group on the server running PSU?

I’m about to refactor my dev and prod environments to leverage gMSA in order to support the migration to MSSQL and other reasons that aid in security.

thanks,

~ Steve

[Mentat]

Hey @sgaglione_laz,

Some of my research on gMSAs said to add it to the Local Admins group. Based on that, and the below copied from Running as a Service Account - PowerShell Universal, I decided to go ahead and add the gMSA as a local admin on the PSU servers.

To run Universal Automation as a Service Account, and not the local system account, an additional set of permissions are required for the Service Account. Windows requires a distinct set of permissions for the Service Account if it is not in the local Administrators group on the host.

BR,
Rob