Timeout setting 'Job Handshake Timeout' is not used it seems

[Steven]

It seems that the property/setting Job Handshake Timeout is not respected or applied, I get timeouts on my jobs:

Error executing job: PowerShell did not respond in a timely fashion. PowerShell Universal is currently waiting 5 seconds for PowerShell to start running the script.

It is the 5 seconds that is too little and yet the settings I do do not help at all.

You can increase the timeout by setting the ‘Job Handshake Timeout’ setting in the Settings \ General \ Automation page.

I have set this in the UI of PSU (Settings / General / Automation / Job Handshake Timeout)

I also have an appsettings.json file copied to various locations (as it is not clear to me where this file should be located):

/config/appsettings.json
/config/.PowerShellUniversal/appsettings.json

The content / setting for Job Handshake Timeout :

  "UniversalAutomation": {
    "Queues": [],
    "JobHandshakeTimeout": 25,
    "JobDebugging": false,
    "ContinueJobOnServerStop": false

The values I put in the diverse appsettings.json files is random as I wanted to see which one would reflect in the UI. None do, no value I set in that json file is showing in the UI.

It is a mystery to me where to put this value so it a) sticks and b) is actually used.

The reason I need to set this to a higher value is that I need to start jobs with a different environment (7.3.4) as the Default environment is 7.2.7 (.Net6) and I have a library that needs 7.3.4 (.Net7)

I have no issue waiting a bit longer because I need to have PSU start an environment outside the Default one but right now I get this timeout at a great rate and it is getting in the way.

EDIT: I also noticed since day one that loading PSU is a bit slow. It seems to ‘hang’ on this step for 5 minutes and then PSU is started:

PowerShell Universal is loading...
Loading configuration files...

Could it be that PSU tries to load configuration files but fails?

Product: PowerShell Universal
Version: 3.9.2

[adam]

As for the job handshake timeout, it looks like it’s loading the wrong appsettings.json file. I do see an issue where the setting is set to 5 in appsettings.linux.json. This is likely causing it to ignore the value you are setting. This is the first time I’ve seen Linux fail to start PS fast enough.

Can you check the installation directory for this file and attempt to change it to a large value?

As for PSU starting up slowly: That’s a bit surprising on Linux as well. It’s usually much snappier than Windows. You could increase the log level from error to debug to see exactly which script it’s attempting to load at this point.

[Steven]

I think you need to point me to where which file should be and what to set. I have tried just about anything and cannot change this behaviour in timeout.
And starting Universal and waiting 5 minutes (for real) is getting rather old and does not make a good debugging platform. Right now I am poking in a lot of places and I have no clue if that is right, did I set the right things, is it picked up by PSU, do I need to restart container or re-create it, etc…
And after all the settings I tried, never ever was it
a) reflected in the UI
b) adhered to the value other than 5 seconds in executing jobs
(even the value from the UI which I have to assume is the right place to change that value)

Sofar I tried these locations and files

[from container perspective]

* /home/Universal/appsettings.linux.json
* /home/Universal/appsettings.json

[from persistent volume/docker host perspective]

* /persistent_volume/.PowerShellUniversal/appsettings.json

And of course in the UI of PSU itself

Please help me out on where to set this value, it is driving me nuts

And regarding the slow startup of PSU, I have set things to Debug for logging, what do you need to investigate this? The startup speed is always the same, even when I only run PSU as the only container. Something has the be blocking but I have no idea what it is. Atm I just accept the 5 minutes startup time as this is not a daily or recurring task normally but during development and restart-prone discovery phase it is super limiting and I notice I am starting to develop a ‘just forget about it’ attitude.
Probably my issue or fault but I have only so much time in a day, sorry

[adam]

Can you remind me how you running this docker container? Starting the empty PSU docker container on my windows host takes less than 5 seconds so I want to make sure I have all the differences accounted for.

In terms of the job setting:

A way to avoid having to set files on disk would be just to use an environment variable.

Here’s how I can run it in my environment to set that value.

docker run --rm -it -e UniversalAutomation__JobHandshakeTimeout=30 -v C:\src\psu-docker:/root -p 5000:5000/tcp ironmansoftware/universal:latest

If you want to double check the config setting is being populated properly, you can use this script. Run it within the integrated environment. It will output which setting is being used and what all the timeouts are set to.

Write-Host ("Env Var: " + $Env:UniversalAutomation__JobHandshakeTimeout)

$configService = [PowerShellUniversal.Client].GetProperty("_config", ([System.Reflection.BindingFlags]::Instance -bor [System.Reflection.BindingFlags]::NonPublic )).GetValue($UniversalClient)
$ConfigSetting = $configService.GetSetting(([PowerShellUniversal.ConfigurationSetting]::JobHandshakeTimeout))
Write-Host "Config Setting: $ConfigSetting"

$database = [PowerShellUniversal.Client].GetProperty("_database", ([System.Reflection.BindingFlags]::Instance -bor [System.Reflection.BindingFlags]::NonPublic )).GetValue($UniversalClient)
$dbSetting = $database.Settings.Get().JobHandshakeTimeout
Write-Host "Database Setting: $dbSetting"

if ($ConfigSetting -eq 30)
{
    "Using Database setting: $dbSetting"
}
else 
{
    "Using Config setting: $ConfigSetting"
}

With the above docker command line, it looks like this.

[information] Env Var: 30 
[information] Config Setting: 30 
[information] Database Setting: 30 
[information] Using Database setting: 30 

[Steven]

This is my compose file.
Note: I set ‘deploy’ as I wanted to see if assigning resources to containers would fix the issue of having a hard time starting the 7.3.4 Environment in a job. The slowness was before this setting just the same.

version: "3.8"
services:
  universal:
    container_name: universal
    image: ironmansoftware/universal:latest
    platform: linux/amd64
    restart: always
    deploy:
      resources:
        limits:
          cpus: '1'
          #memory: 500M
        reservations:
          cpus: '0.10'
          #memory: 20M
    volumes:
      - ./universal-data:/root
      - type: bind
        source: /media/storage
        target: /media-storage
      - type: bind
        source: /home/steven/docker
        target: /docker-storage
      - type: bind
        source: /mnt/shield
        target: /shield
    ports:
      - 8082:5000
    extra_hosts:
      - "host.docker.internal:host-gateway"

So should I set this in the compose file then?

environment:
  UniversalAutomation__JobHandshakeTimeout: 30

I mean, in the compose file is OK but seeing this is a setting of PSU specifically and there is a settings file just for it, I would say that the primary source is the json file meant for it and a fallback could be this environment setting in the docker compose file.

Please let me know if you need anything else to debug this setting and the general slowness of PSU.

EDIT: I just ran the same code you posted to show all the values. I did this after adding the environment variable setting in my compose file and re-starting the container.

[information] Env Var: 30 
[information] Config Setting: 30 
[information] Database Setting: 15 
[information] Using Database setting: 15

Indeed, the UI shows 15 still. Which one will actually be used? The DB one? And what is the proper way to set this once and for all, no differences and guesswork, just me setting that value, restart PSU and move on. There has to be a way to set this in a predictable way, right?

[adam]

I’ll play with this compose file and see if I can reproduce the slowness during start. As for the job handshake timeout setting, use the one in the UI (database). The example you show above is using the database setting of 15. It should be the goto and the only reason you are having problems is the issue with the current appsettings.linux.json file we ship with the PSU image. We will get that corrected in the next version.

[Steven]

Ok, lets see.
I guess the “set it and forget it” is not going to happen via config file(s) but needs to be done manually via the UI after starting PSU. The persistent folder should make that setting last I hope.
I would have thought that setting the environment variable would overrule all other similar settings, at least that is what I am used to from various other systems.

FYI: I set this UI value many, many times and when I get a job timeout, it is always telling that the 5 second limit was reached and I had it set to 27 and I can tell it did not wait for 27 seconds before the timeout came.

Good Luck!

[adam]

If you want to set it and forget it, you can set the env var to something more than 30.

environment:
  UniversalAutomation__JobHandshakeTimeout: 90

One reason we have the setting in the UI\database is that you can change it in one place and it will be picked up by all PSU nodes in a multi-instance configuration rather than having to change env var\files on each node.

Also, the setting you set in the UI should be persisted to the database so if you change it there, you shouldn’t have to change it again after restart.

[adam]

I adjusted the compose file a bit to simplify it and it still starts really fast for me.

version: "3.8"
services:
  universal:
    container_name: universal
    image: ironmansoftware/universal:latest
    platform: linux/amd64
    restart: always
    volumes:
      - ./universal-data:/root
    ports:
      - 5000:5000

We might need to review what’s currently in your repo. Feel free to respond to the support case you opened and we can setup some time to take a look together.

[Steven]

Thanks Adam!
I will respond in the case this evening and we can discuss from there.
Biggest difference I see for now would be the port?

[adam]

Thanks.

I switched back to 8082 and it still is quick so that doesn’t seem to be it.