Jobs Queued but not Running

Looks like Hangfire has given up. I have over 8000 jobs queued since the weekend (almost all Git sync) and no more jobs will run:

I’ll restart the app service to see if that gets it going, but thought I’d better report this. Anything I can do to diagnose what got us into this state?

Product: PowerShell Universal
Version: 1.4.6

OK I’ve restarted the app service and can see the number of queued jobs slowly dropping as PSU processes them. Gonna take a while before it gets to this morning’s scheduled jobs. :frowning:

It seems like the git sync job was stuck. We’ve actually made some improvements to this in 3.3 to avoid this issue. As multiple git sync jobs won’t be able to run on the same machine at once, which appears to be happening.

We’ve also seen that git sync can hang for a long time when trying to commit large changes (think installing the Az module into the repo). This is wayyyy worse if your repo is on mounted Azure storage rather than directly within the web app’s local storage.

1 Like