Script run fails when using gMSA for PSU service

wakeke · November 8, 2021, 11:32am

When using a gMSA to run PowerShell Universal process under, script execution doesn’t work if using different credentials.
This have been testet with a PowerShell v7.1.5 enviroment (persistent).
Debug logging was also enabled for every LogLevel entry, but that didn’t seem to give more logs for some reason.

A script was created to run under the PS7 enviroment with custom credentials.

When running the script the error “Error executing job: Did not receive port from client process.” was recieved and there is no indication of the error in the Application Event Log or the PSU log.

But the PowerShellCore event log show the following Warning, which indicates that the “User: domain\StandardSVC” doesn’t have access to PSU’s gMSA temp folder.

Error Message = The specified drive root "C:\Users\gMSA~1\AppData\Local\Temp\" either does not exist, or it is not a folder.
Fully Qualified Error ID = DriveRootError

Provider name = FileSystem

Context:
        Severity = Warning
        Host Name = ConsoleHost
        Host Version = 7.1.5
        Host ID = 085c49fe-f3f4-41d9-a98b-fe53f8131252
        Host Application = C:\Program Files\PowerShell\7\pwsh.dll -NoProfile -Command & { [System.Reflection.Assembly]::LoadFrom('C:\Program Files (x86)\Universal\Host\host.dll') | Out-Null; [UniversalHost.AgentService]::StartJob(59361, 6644, $True, 59312) }
        Engine Version = 
        Runspace ID = 
        Pipeline ID = 
        Command Name = 
        Command Type = 
        Script Name = 
        Command Path = 
        Sequence Number = 4
        User = domain\StandardSVC
        Connected User = 
        Shell ID = Microsoft.PowerShell

The account ‘domain\StandardSVC’ was then assigned full access to the gMSA’s user profile folder.

Trying to run the script again a new error was recieved “Error executing job: Access is denied.” and i cannot find anything regarding the “Accesss is denied” error in
In the gMSA temp folder empty files is created called “psu..txt” and every new files a created about every 15 seconds.
Task Manager shows more than one new pwsh.exe processes getting started and stopped shortly after. It looks like it wouldn’t stop before i restarted the PSU service.

In the PSU log we now see the following error multiple times:

2021-11-08 10:41:11.686 +01:00 [ERR] Error in ConcurrentJobFilter
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.TryPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 71
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.OnPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 34
2021-11-08 10:41:26.690 +01:00 [ERR] Error in ConcurrentJobFilter
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.TryPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 71
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.OnPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 34
2021-11-08 10:42:01.960 +01:00 [ERR] Error in job thread
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.ExecutionService.<>c__DisplayClass19_0.<<ExecutePowerShell>b__4>d.MoveNext() in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ExecutionService.cs:line 584
2021-11-08 10:42:11.720 +01:00 [ERR] Error in ConcurrentJobFilter
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.TryPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 71
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.OnPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 41
2021-11-08 10:42:11.720 +01:00 [ERR] Error in ConcurrentJobFilter
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.TryPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 71
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.OnPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 41
2021-11-08 10:42:16.956 +01:00 [ERR] Error in ConcurrentJobFilter
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.TryPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 71
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.OnPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 41
2021-11-08 10:42:31.965 +01:00 [ERR] Error in ConcurrentJobFilter
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.TryPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 71
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.OnPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 41
2021-11-08 10:42:41.733 +01:00 [ERR] Error in ConcurrentJobFilter
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.TryPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 71
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.OnPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 41
2021-11-08 10:42:56.740 +01:00 [ERR] Error in ConcurrentJobFilter
System.NullReferenceException: Object reference not set to an instance of an object.
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.TryPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 71
   at UniversalAutomation.Services.Automation.ConcurrentJobFilter.OnPerforming(PerformingContext filterContext) in D:\a\universal\universal\src\UniversalAutomation\Services\Automation\ConcurrentJobFilter.cs:line 41

The PowerShellCore event log also indicate that new pwsh processes is started but there is no errors or warnings.
The Security event log only shows audit success for the “domain\StandardSVC” user account so the access denied doesn’t look to be permissions related and the account already have ‘logon as a batch job’ permission.

If I change the PSU service to run under the LocalSystem account the issue ins’t there.
I haven’t tried to run the PSU service under a standard user account, so i don’t know if it’s only a problem with gMSA.

Am i doing something wrong or missing something?

Other issues I noticed when trying to debug this issue:

Retry limit cannot be changed for a script after it was created (the setting doesn’t save)
Editing a variable doesn’t show the current value in the edit box.
Editing an enviroment doesn’t show current value of “persistent runspace”. The ‘checkbox’ is always disabled.
After restart of the PSU service a Dashboard shows as “Stopped” even though it’s started. Pressing start button resolves this until next service restart.

I haven’t got a GitHub account so i cannot post it there.

Product: PowerShell Universal
Version: 2.4.1

PorreKaj · November 8, 2021, 12:32pm

Same goes for IIS, the behaviour is mentioned in the docs.

IIS Limitations with Universal Automation

App Service configured as Local System - Scripts will execute as the System Account by default and a Run as Accounts CAN be specified when executing a Script in Universal Automation

App Service configured as a Service Account - Scripts can ONLY be executed with the Service Account and a ****** Run as Account **** CANNOT** be specified when executing scripts.

IIS - choosing-an-app-pool-identity - PowerShell Universal

Edit. Mentioning this purely for info btw.

wakeke · November 8, 2021, 12:38pm

Hi PorreKaj

But this is Kestrel and not IIS. The account can be used as long as it has permissions for the gMSA’s temp folder it authenticates fine. The doc doesn’t mention this about Kestrel.

PorreKaj · November 8, 2021, 12:43pm

I know, I just mentioned it because I think its related.

adam · November 8, 2021, 5:27pm

I wonder if it’s failing to connect to the database. Try setting the permissions on the C:\ProgramData\UniversalAutomation folder for the gMSA so it has full permissions.

I think the file in particular it may be failing to access is C:\ProgramData\UniversalAutomation\database.db.

wakeke · November 8, 2021, 7:33pm

Hi Adam

The MSA already have full access to the database since the PSU service won’t start without it.
I tried to give the standard service account the same permissions but that doesn’t help either.
Also tried to make it administrator and give it extra user rights assignment permissions without any luck.

The access denied has to come from somewhere.
Is there any way to enable more debugging info than adding this?

    "LogLevel": {
      "Default": "Debug",
      "Microsoft": "Debug",
      "Microsoft.Hosting.Lifetime": "Debug",
      "Grpc": "Debug"
    }

wakeke · November 8, 2021, 8:15pm

I tried giving the gMSA local administrators access on the server and then it worked.
This means that it must be some local security policy permission that it needs.

EDIT: Actually both the gMSA and standard SA needed local administrator permissions for it to work.

I noticed using Process Monitor that Universal.Server.exe try to create a reg key under different
location all havening something to do with system certificates.

adam · November 8, 2021, 11:23pm

I have no idea what’s causing this so I’ll open an issue for it. It likely has to do with some ASP.NET Core thing we have configured and not something we are doing directly so it might take a bit of research to figure out what’s causing that.

wakeke · November 9, 2021, 7:43am

Yeah you could be right. I have tried every combination of permissions but only adding both acccounts to local administrators is working.

I have tried assigning them both all of the following user rights with have no effect.

Impersonate a client after authentication
Obtain an impersonation token for another user in the same session
Allow log on locally
Act as part of the operating system
Replace a process level token
Adjust memory quotas for a process
Log on as a service
Log on as a batch job

The access denied message that Process Monitor found i gave them access to with no effect.
It looks like every PowerShell session (doesn’t matter which editions or if it’s the Universals build in) tries to access a GitHub DLL in the Universal host folder but they can’t.

This screenshot is for the standard SA but the gMSA also trieds to do this.

If you need any information to fix this issue then please let me know

wakeke · December 15, 2021, 3:51pm

Hi @adam

Is this fixed in 2.6.0?

Or is this another issue?
“Fixed an issue where running jobs under alternate credentials could result in an Access Denied error”

Topic		Replies	Views
"Error executing job: Did not receive port from client process." when using non-Admin Run As Account PowerShell Universal	4	487	January 6, 2022
PSU service not able to start using GMSA PowerShell Universal	1	325	August 31, 2022
Problems using gMSA account for PSU service - Cert error PowerShell Universal	16	2094	November 4, 2022
Scripts sometimes fail with - Error executing job: Did not receive port from client process PowerShell Universal	5	312	February 15, 2023
Error with credentials and schedules.ps1 PowerShell Universal	5	474	May 3, 2022

Script run fails when using gMSA for PSU service

Related topics