Jobs failing most of the times

PowerShell Universal
1

When I schedule a script hourly, most of the time the job is failing with the following error:

Error executing job: PowerShell did not respond in a timely fashion. PowerShell Universal is currently waiting 30 seconds for PowerShell to start running the script.
You can increase the timeout by setting the ‘Job Handshake Timeout’ setting in the Settings \ General \ Automation page.
If you are still experiencing issues, you may need to consider increasing the resources available on your system or take advantage of the Random Delay parameter for schedules to stagger jobs.

I run PowerShell Universal on IIS and have applied all the settings which are documented here IIS - PowerShell Universal

The Powershell.exe process is running in the background while triggering the job. The following error is logged in the logs.

[11:41:03 INF] Error executing job 69: PowerShell did not respond in a timely fashion. PowerShell Universal is currently waiting 30 seconds for PowerShell to start running the script.
You can increase the timeout by setting the ‘Job Handshake Timeout’ setting in the Settings \ General \ Automation page.
If you are still experiencing issues, you may need to consider increasing the resources available on your system or take advantage of the Random Delay parameter for schedules to stagger jobs.
at UniversalAutomation.Services.GrpcExecutionService.ExecuteScript(Job job, ExecutionEnvironment environment, ExecuteScriptRequest request, ExecutionCallback executionCallback) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Automation\GrpcExecutionService.cs:line 113
at UniversalAutomation.ExecutionService.ExecutePowerShell(Script script, IEnumerable`1 parameters, Job job, Schedule schedule, ExecutionCallback callback) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 770
at UniversalAutomation.ExecutionService.TryExecute(Job job, Schedule schedule) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 406
at UniversalAutomation.ExecutionService.Execute(Job job, Schedule schedule) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 227 System.Exception

Maybe off-topic but I also see the following error a lot of time:

[11:42:06 INF] CORS policy execution failed.

It is not a system resource problem.
Sometimes the job is finished without any error.

It’s a very basic script, for testing purposes:
Get-ADUser -Identity “r.roodenburg”

Anyone else seeing this issue?

Some facts

  • PowerShell Universal 4.2.10
  • Windows Server 2022
  • PowerShell 5.1 environment
  • While the Application Pool is running with the LocalSystem identity, the script is running with another user (secret variable)
2

FWIW, I know you say it’s not a resource issue, but when I faced that error message, I added another vCPU and a few GB of RAM and it stopped occurring. The server itself never really showed any signs of resource contention (i.e., at the OS level or in our monitoring tools), but after adding the additional resources the PowerShell did not respond in a timely fashion. errors went away.

3

Hi @OtherPeoplesCode,

I really appreciate your time for answering my question. May I ask what kind of sizing you have for your VM?

Kinds regards,
Rico

4

The VM is 4 vCPU and 16GB of RAM. Before that, it was 2 vCPU and 8GB RAM when we were running into the issue. For reference, this instance has a couple dashboards, and runs ~2.5k ‘jobs’ per day (we have couple small jobs that run every few minutes).

5

Thanks for sharing! Will test it soon.
Weird enough, with PowerShell 7.4 the same script is working fine everytime it runs.

6

Not sure if it will help or is related but I had a similar question/issue as you it seems, here is the thread:

Timeout setting ‘Job Handshake Timeout’ is not used it seems