Large Amount of Job Errors Post 3.7.4 Upgrade

PowerShell Universal
1 
Product: PowerShell Universal
Version: 3.7.4

My Team recently upgraded our PS Universal from 3.5.5 to 3.7.4, which has caused a lot of job failure issues.

I am receiving in about every other job execution, Did not receive port from client process. Previously this was a rare maybe once a day issue.

These jobs are running using the non-integrated Powershell 5.1 instance. The only changes in our environment are that we have a custom module for a few SQL related commands.

Jobs kick off, and then fail with the above mentioned error.

I’ve increased the job timeout timer to 240 seconds, but unfortunately this does not help the issue, and only increases the time it takes to fail.

Viewing Diagnostics under general settings shows some items that may be related:

[1673639946925] - AuthenticationScheme: "Cookies" was successfully authenticated.
[1673639946933] - AuthenticationScheme: "OpenIdConnect" was successfully authenticated.
[1673639946940] - Attempting to bind parameter '"id"' of type '"System.Int64"' using the name '"id"' in request data ...
[1673639946950] - Done attempting to bind parameter '"id"' of type '"System.Int64"'.
[1673639946961] - Attempting to bind parameter '"timestamp"' of type '"System.Boolean"' using the name '"timestamp"' in request data ...
[1673639946970] - Done attempting to bind parameter '"timestamp"' of type '"System.Boolean"'.
[1673639946979] - Attempting to bind parameter '"format"' of type '"System.String"' using the name '""' in request data ...
[1673639946987] - Could not find a value in the request with name '""' for binding parameter '"format"' of type '"System.String"'.
[1673639946995] - Done attempting to bind parameter '"format"' of type '"System.String"'.
[1673639947208] - AuthenticationScheme: "Cookies" was successfully authenticated.

as well as

[1673640129308] - Finished groom job.
[1673640131241] - Error executing job 2982: Did not receive port from client process.    at UniversalAutomation.Services.GrpcExecutionService.ExecuteScript(Job job, ExecutionEnvironment environment, ExecuteScriptRequest request, ExecutionCallback executionCallback) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\GrpcExecutionService.cs:line 87
   at UniversalAutomation.ExecutionService.ExecutePowerShell(Script script, IEnumerable`1 parameters, Job job, Schedule schedule, ExecutionCallback callback) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 719
   at UniversalAutomation.ExecutionService.TryExecute(Job job, Schedule schedule) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 368
   at UniversalAutomation.ExecutionService.Execute(Job job, Schedule schedule) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 215 System.Exception
[1673640131351] - AuthenticationScheme: "Cookies" was successfully authenticated.

I’ve been unable to find a probable cause for why this may be occuring. My only theory is possibly the custom module, but that worked fine in 3.5.5. Any help would be greatly appreciated, as this has had a significant impact on automation efforts.

2

If I had to guess, there is some sort of assembly conflict happening and it’s failing to load the custom host for PSU.

Try this in a WinPS console:

Import-Module C:\Program Files (x86)\Universal\Host\Host.dll 
[UniversalHost.AgentService]::StartJob(0, 0, $true, 0)

You’ll see errors since it won’t be able to actually start a job but it will help to figure out if it is failing to load an assembly.

3

Hi Adam, thanks for the response.

I ran the command you sent from an elevated admin ISE window, and a non elevated ISE window from the credentials of one of my ‘run as’ accounts. Both sat indefinitely with no output.

I also ran the command from an ISE window, but changed from the module path from 'C:\Program Files (x86)\Universal\Host\Host.dll'

to: C:\Program Files (x86)\Universal\Host.dll as I have a host.dll file in that location as well. This met me with the following errors:

Import-Module : Could not load file or assembly 'System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51' or 
one of its dependencies. The system cannot find the file specified.
At line:1 char:1
+ Import-Module 'C:\Program Files (x86)\Universal\Host.dll'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], FileNotFoundException
    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.ImportModuleCommand
 
Exception calling "StartJob" with "4" argument(s): "Could not load file or assembly 'System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral, 
PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified."
At line:2 char:1
+ [UniversalHost.AgentService]::StartJob(0, 0, $true, 0)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : FileNotFoundException

This may be expected since I changed the path, but wanted to mention it.

I also ran the command you sent from the account that runs the PS universal service, but was met with this error:

Import-module : Could not load file or assembly 'System.Threading.Tasks.Extensions, Version=4.2.0.1, Culture=neutral,
PublicKeyToken=cc7b13ffcd2ddd51' or one of its dependencies. The system cannot find the file specified.
At line:1 char:1
+ Import-module 'C:\Program Files (x86)\Universal\Host\Host.dll'
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Import-Module], FileNotFoundException
    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.ImportModuleCommand
4

The Host folder is specifically for Windows PowerShell so that might be why you are seeing the error with the host.dll in the root folder.

If the command I sent was just hanging, that means it’s not necessarily an assembly loading problem.

If you’d like to rollback and would like a build that contains the CVE fix for 3.5, we can get that queued up. This will give us time to diagnosis this issue and get you back up and running.

Can you try to enable job debugging in appsettings.json? It will produce txt files in the temp folder with the job ID and prefixed with psu. It logs some raw console output during the startup of jobs.

5

Hey Adam,

I’ve enabled debug mode, now of course everything appears to be working correctly for the time being.

When a failure does occur, where do these logs files get sent? I tried checking the service accounts AppData folder, and install locations for PSU but could not find the temp folder you mentioned.

I’d like to avoid rolling back if possible, as we’re using some of the fixes introduced in these newer versions, but if this weekend goes poorly, I will let you know on Monday that we’d like to roll back.

6

Sounds good. We use the %TEMP% environment variable for the service account to determine where to store those log files. They should show up in failure and successful jobs. You should be able to use runas to start a pwsh prompt and evaluate what the $ENV:Temp path is for that user.

7

Hey Adam,

After letting it sit this weekend + holiday, we still had failures happening quite often. I checked this temp repository you mentioned, and unfortunately do not see any job logs. Both at the $ENV:Temp folder, and its root.

image
image633×589 47.2 KB

The root contains log filles I previously manually downloaded, but nothing beyond that.

I verified debug mode is set in c:\program files x86\Universal\Appsettings.json

image

Any ideas as to next possible steps? Or perhaps I just messed something up?

Note: I set debug mode to true in both appsettings.json and today set it to true also in appsettings.desktop.json. Still no logs unfortunately.

8

For any future browsers of this thread, I resolved this issue by increasing the hardware alloted to the VM PSU was running on.

Previously, it was running on dual core 8GB RAM, with unlimited CPU usage. Increasing to quad core & 16GB RAM cleared the issues my team was seeing.