Scheduler not running/working

PowerShell Universal
1

My scheduler is for some reason not working, and I’ve no idea how to fix it!

Recent changes/events that may or may not have anything to do with it breaking:

I upgraded to 5.6.1 (though I think the issue may have been present before this point - I’m not sure) I purged my DB and did a clean install also.

I configured one-way git (i noticed the create schedule button is gone / one-way sync forces read only mode) so i figured this could have caused an issue, I’ve since removed all git settings and rebooted but no joy.

A (different) manually run job got stuck in ‘cancelling’ state, not sure why, but I restarted the service & server, it stayed in ‘cancelling’ In the end i hit archived to see if that would do anything, not sure if it’s still there in the background or not (not sure how to view archived jobs or if it’s as good as gone?)

“Groom Job Status” is showing as failed on both my nodes, no idea why, I do have some suspect errors in my error log, showing a ‘fatal error starting psu’ though it seemingly starts up fine for me to access the admin console and apps, and run scripts manually. I can see the schedule execution trying, but it complains about not being able to find it.

2025-07-29 17:38:29.421 +01:00 [FTL][] Fatal error starting PowerShell Universal.
System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Collections.Concurrent.BlockingCollection`1[[System.String, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]'.
   at Universal.Server.Services.Configuration.ConfigurationSystemWatcher.ProcessNotificationQueue(CancellationToken cancellationToken) in D:\a\universal\universal\src\PowerShellUniversal.Configuration\ConfigurationSystemWatcher.cs:line 120
   at Universal.Server.Services.Configuration.ConfigurationSystemWatcher.<StartAsync>b__18_0() in D:\a\universal\universal\src\PowerShellUniversal.Configuration\ConfigurationSystemWatcher.cs:line 56
   at Universal.Server.Services.Configuration.ConfigurationSystemWatcher.StopAsync(CancellationToken cancellationToken) in D:\a\universal\universal\src\PowerShellUniversal.Configuration\ConfigurationSystemWatcher.cs:line 80
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Universal.Server.Program.Main(String[] args) in D:\a\universal\universal\src\Universal.Server\Program.cs:line 37
2025-07-29 18:05:05.133 +01:00 [ERR][PowerShellUniversal.Automation.ExecutionService] Schedule <ScheduleName> not found.
2

I did notice something I thought was odd - in the database, computers table, nodeA and NodeB both have a GitBranch, GitSettings Type and EditMode all populated, editmode is 0 for both.

However in the UI for both instances git settings were deleted and show ‘Git Synchonization is not configured’.

Should these fields have been set to NULL when i deleted my git config or is this a red herring?

Edit: Given that I believe this issue was present before I reinstalled and purged my DB, thinking about it now, I reconfigured the git sync afterwards, so I’m wondering now if this is the cause. Even though it’s a pain, I may try a purge of the DB tomorrow and I won’t touch any git sync settings to test that theory.

Edit2: stopped the service on both nodes, purged the DB, restarted node one, then node two, attempted a new schedule and still no joy. So it’s not what I thought, still non the wiser.

3

We are experiencing a sudden issue in all environments where if we use a computer group schedules won’t run those jobs. It queues in hangfire and doesn’t show up in PSU jobs. Setting schedules to run on the computer name fixes it. I’ll have to see if we have that specific error in logs as well.

4

Thanks for the input - I use computer groups too, I think there’s potentially a few different things going on in my situation, potential issues, but also in part my understanding of how schedules work across computers setup to use the same PSU database too.

Originally when I had my issue, all schedules were broken.

I have 2 computers using the same SQL database. I tested creating a schedule on computerA and executing on ComputerA, then creating on computerA executing on computerB and so on until i’d gone through all variations. Nothing was working.

After purging my DB last night, existing schedules are now running on both hosts, and new schedules are running on the host that they’ve been created on.

I realize that schedules are defined in code, so if the schedule is git synced on both computers, then it’s fine. If it’s a new schedule and only on one computer it will fail on the other because it doesnt exist. (I was using one time schedules created from an app page as part of an automation).

I noticed there’s a schedules sql table, but this doesnt seem to be used, it would be good if we could push the schedules to sql rather than the schedules.ps1 file as I think that would solve my problem, but for now, whatever caused my original issue is resolved, and my understanding on syncing the schedules.ps1 file is a little better, so I’m okay for now. I’m just running my one time schedules on the same node, since they clean themselves up after running anyway, but this throws a spanner in the works with how i want to use git sync - so i’ll leave that for now and continue doing that manually for the time being.

I do still have my groom job failures though - havent got to the bottom of that one yet

1 Like
5

Glad you figured it out! I figured out the issue on my side I think, not why it was working and what triggered it to just stop, but the root had to do with hangfire.hash table not being cleaned up through version upgrades and old queues persist.

6

Take a look here:

search for “Database Persistence for Resources”
You can actually change from file to database.

7

Probably should have updated here, but thanks for linking this too.

I updated a github issue and was discussing this, adam mentioned that one time schedules are by default pushed to the database anyway, but I did not notice this behavior in a HA/SQL setup (couldnt see the schedules either in schedules.ps1 or database) and they were failing when creating on computerA and executing against computerB (otherwise working on the same node).

After changing database persistence, this worked as expected and resolved my issue - one time schedules are working across nodes and appearing in sql.

However my groom job failures are still occuring - no idea why.

1 Like
8

Just as an update for anyone wondering, I raised a support ticket and the groom job failures are a known issue on 5.6.1 a planned fix will be coming in 5.6.2

1 Like