Scheduled jobs don't run

PowerShell Universal
1 
Product: PowerShell Universal
Version: 2.8.2

Noticed that every few days my jobs won’t run, no failures are shown in the job log, they just don’t run. If I try to manually run a job in this state it sits in the Queued status and never runs. I see this over and over in the logs as well, not sure if it related?

2022-02-21 21:14:57.874 -05:00 [DBG] Connection id "0HMFJUEPJNK97" accepted.
2022-02-21 21:14:57.874 -05:00 [DBG] Connection id "0HMFJUEPJNK97" started.
2022-02-21 21:14:57.895 -05:00 [DBG] Connection "0HMFJUEPJNK97" established using the following protocol: "Tls12"
2022-02-21 21:14:57.897 -05:00 [INF] Request starting HTTP/1.1 GET https://10.101.9.49/ - -
2022-02-21 21:14:57.897 -05:00 [DBG] AuthenticationScheme: Cookies was not authenticated.
2022-02-21 21:14:57.897 -05:00 [DBG] No response compression available, the Accept-Encoding header is missing or invalid.
2022-02-21 21:14:57.897 -05:00 [DBG] The request path / does not match a supported file type
2022-02-21 21:14:57.897 -05:00 [DBG] The request path  does not match the path filter
2022-02-21 21:14:57.897 -05:00 [DBG] No candidates found for the request path '/'
2022-02-21 21:14:57.897 -05:00 [DBG] Request did not match any endpoints
2022-02-21 21:14:57.899 -05:00 [INF] Executing ChallengeResult with authentication schemes ([]).
2022-02-21 21:14:57.899 -05:00 [INF] AuthenticationScheme: Cookies was challenged.
2022-02-21 21:14:57.900 -05:00 [INF] Request finished HTTP/1.1 GET https://10.101.9.49/ - - - 302 0 - 2.9154ms
2022-02-21 21:14:57.900 -05:00 [DBG] Connection id "0HMFJUEPJNK97" disconnecting.
2022-02-21 21:14:57.900 -05:00 [DBG] Connection id "0HMFJUEPJNK97" stopped.
2022-02-21 21:14:57.900 -05:00 [DBG] Connection id "0HMFJUEPJNK97" sending FIN because: "The Socket transport's send loop completed gracefully."
2

Can you take a look in the hangfire dashboard to see if there is anything useful? It might have some more details on why the queue is running.

3

I am showing 294 enqueued, and 40 processing. The rest are 0’s

image

Looks like all of the processing jobs for the the git sync

image
image881×315 9.33 KB

image
image879×752 23.2 KB

I have push only sync enabled.

Odd that it says 0 jobs succeeded. I have had scheduled jobs run fine today after I rebooted, unless I am reading this wrong.

4

Happened again yesterday. Seems like all jobs stopped around noon. Checked hangfire before rebooting and I have almost 3k jobs sitting enqueued

image

Most of them seem to be git sync. I do get errors in the console from time to time where git sync fails, but when I check in the settings the most recent git sync looks correct for the last time I saved something. Wondering if something is up with that so I disabled it.

5

It seems to me that the git sync jobs are hanging for some reason and then hangfire is not starting new jobs.

I’ll open an issue to investigate this. If possible, could you please take a memory dump of the process and upload it here?

6

Should I re-enable the git sync before getting a memory dump? Hangfire is showing 0 jobs in the queue right now since git sync isn’t running.

7

This was the settings I had in place for my git sync. We use an on prem Gitlab instance

image
image775×203 5.04 KB

8

I identified an issue that would cause the hangfire jobs to hang like this if one of the git syncs hangs. This has been resolved. It might not prevent the git sync from hanging but it will fix the issue where the jobs stop running and you have like 40 git syncs happening. They will now time out and be released back to the pool so other jobs can continue. The fix for this will be in our nightly build tonight.