I recently upgraded to 4.3.1 from I think 3.2 and had some problems logging in at first but managed to get everything running but now I can’t schedule jobs. It gives the following error:
Error
Invalid configuration: schedules.ps1
Object reference not set to an instance of an object.
07/05/2024 10:38 AM
I’ve tried deleting schedules.ps1 and PSU recreates it and the job IS in schedules.ps1 however universal dashboard is showing next execution as never and it never does run it. Any ideas on how to fix it?
OK I did that and tried scheduling a job. The error is kinda strange it seems it can’t find the america/new-york timezone?
Blockquote
2024-07-08 09:50:04.308 -04:00 [INF] User profile is available. Using ‘C:\Windows\system32\config\systemprofile\AppData\Local\ASP.NET\DataProtection-Keys’ as key repository and Windows DPAPI to encrypt keys at rest.
2024-07-08 09:50:05.661 -04:00 [INF] RPC services being provided by PowerShellUniversal.IGitSettingsService: 4
2024-07-08 09:50:05.726 -04:00 [INF] RPC services being provided by UniversalAutomation.IPublicGitService: 3
2024-07-08 09:50:05.735 -04:00 [INF] RPC services being provided by UniversalAutomation.IPublicGitSyncService: 5
2024-07-08 09:50:06.112 -04:00 [INF] Starting Hangfire Server using job storage: ‘Hangfire.MemoryStorage.MemoryStorage’
2024-07-08 09:50:06.115 -04:00 [INF] Using the following options for Hangfire Server:
Worker count: 100
Listening queues: ‘default’, ‘psuserver’
Shutdown timeout: 00:00:15
Schedule polling interval: 00:00:15
2024-07-08 09:50:06.131 -04:00 [INF] Application started. Press Ctrl+C to shut down.
2024-07-08 09:50:06.131 -04:00 [INF] Hosting environment: Production
2024-07-08 09:50:06.131 -04:00 [INF] Content root path: C:\inetpub\wwwroot
2024-07-08 09:50:06.131 -04:00 [INF] Request starting HTTP/1.1 POST htps/psuserver/notificationhub/negotiate?negotiateVersion=1 text/plain;charset=UTF-8 0
2024-07-08 09:50:06.131 -04:00 [INF] Request starting HTTP/1.1 POST htps/psuserver/terminalhub/negotiate?negotiateVersion=1 text/plain;charset=UTF-8 0
2024-07-08 09:50:06.132 -04:00 [INF] Request starting HTTP/2 POST htps/psuserver/notificationhub/negotiate?negotiateVersion=1 text/plain;charset=UTF-8 0
2024-07-08 09:50:06.142 -04:00 [INF] Server psuserver:9952:e646abbd successfully announced in 14.5983 ms
2024-07-08 09:50:06.145 -04:00 [INF] Server psuserver:9952:e646abbd is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, ExpirationManager, CountersAggregator, Worker, DelayedJobScheduler, RecurringJobScheduler…
2024-07-08 09:50:06.181 -04:00 [INF] PowerShell Universal Version: 4.3.1
2024-07-08 09:50:06.182 -04:00 [INF] OS Version: Microsoft Windows NT 10.0.14393.0
2024-07-08 09:50:06.182 -04:00 [INF] OS Architecture: x64
2024-07-08 09:50:06.182 -04:00 [INF] Process Architecture: x64
2024-07-08 09:50:06.182 -04:00 [INF] Node Name: psuserver
2024-07-08 09:50:06.226 -04:00 [INF] Server psuserver:9952:e646abbd all the dispatchers started
2024-07-08 09:50:06.198 -04:00 [ERR] Connection ID “18158513707758413958”, Request ID “80006887-0002-fc00-b63f-84710c7967bb”: An unhandled exception was thrown by the application.
System.InvalidOperationException: Provide Authority, MetadataAddress, Configuration, or ConfigurationManager to OpenIdConnectOptions
at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectOptions.Validate()
at Microsoft.AspNetCore.Authentication.RemoteAuthenticationOptions.Validate(String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationBuilder.<>c__DisplayClass4_02.<AddSchemeHelper>b__1(TOptions o) at Microsoft.Extensions.Options.ValidateOptions1.Validate(String name, TOptions options)
at Microsoft.Extensions.Options.OptionsFactory1.Create(String name) at Microsoft.Extensions.Options.OptionsMonitor1.<>c.b__10_0(String name, IOptionsFactory1 factory) at Microsoft.Extensions.Options.OptionsCache1.<>c__DisplayClass3_11.<GetOrAdd>b__2() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)
— End of stack trace from previous location —
at System.Lazy1.CreateValue() at Microsoft.Extensions.Options.OptionsCache1.GetOrAdd[TArg](String name, Func3 createOptions, TArg factoryArgument) at Microsoft.Extensions.Options.OptionsMonitor1.Get(String name)
at Microsoft.AspNetCore.Authentication.AuthenticationHandler1.InitializeAsync(AuthenticationScheme scheme, HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT1.ProcessRequestAsync()
2024-07-08 09:50:06.198 -04:00 [ERR] Connection ID “17870283335901651706”, Request ID “800022fb-0003-f800-b63f-84710c7967bb”: An unhandled exception was thrown by the application.
System.InvalidOperationException: Provide Authority, MetadataAddress, Configuration, or ConfigurationManager to OpenIdConnectOptions
at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectOptions.Validate()
at Microsoft.AspNetCore.Authentication.RemoteAuthenticationOptions.Validate(String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationBuilder.<>c__DisplayClass4_02.<AddSchemeHelper>b__1(TOptions o) at Microsoft.Extensions.Options.ValidateOptions1.Validate(String name, TOptions options)
at Microsoft.Extensions.Options.OptionsFactory1.Create(String name) at Microsoft.Extensions.Options.OptionsMonitor1.<>c.b__10_0(String name, IOptionsFactory1 factory) at Microsoft.Extensions.Options.OptionsCache1.<>c__DisplayClass3_11.<GetOrAdd>b__2() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)
— End of stack trace from previous location —
at System.Lazy1.CreateValue() at Microsoft.Extensions.Options.OptionsCache1.GetOrAdd[TArg](String name, Func3 createOptions, TArg factoryArgument) at Microsoft.Extensions.Options.OptionsMonitor1.Get(String name)
at Microsoft.AspNetCore.Authentication.AuthenticationHandler1.InitializeAsync(AuthenticationScheme scheme, HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT1.ProcessRequestAsync()
2024-07-08 09:50:06.198 -04:00 [ERR] Connection ID “17798225728978813756”, Request ID “8000033d-0000-f700-b63f-84710c7967bb”: An unhandled exception was thrown by the application.
System.InvalidOperationException: Provide Authority, MetadataAddress, Configuration, or ConfigurationManager to OpenIdConnectOptions
at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectOptions.Validate()
at Microsoft.AspNetCore.Authentication.RemoteAuthenticationOptions.Validate(String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationBuilder.<>c__DisplayClass4_02.<AddSchemeHelper>b__1(TOptions o) at Microsoft.Extensions.Options.ValidateOptions1.Validate(String name, TOptions options)
at Microsoft.Extensions.Options.OptionsFactory1.Create(String name) at Microsoft.Extensions.Options.OptionsMonitor1.<>c.b__10_0(String name, IOptionsFactory1 factory) at Microsoft.Extensions.Options.OptionsCache1.<>c__DisplayClass3_11.<GetOrAdd>b__2() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode)
— End of stack trace from previous location —
at System.Lazy1.CreateValue() at Microsoft.Extensions.Options.OptionsCache1.GetOrAdd[TArg](String name, Func3 createOptions, TArg factoryArgument) at Microsoft.Extensions.Options.OptionsMonitor1.Get(String name)
at Microsoft.AspNetCore.Authentication.AuthenticationHandler1.InitializeAsync(AuthenticationScheme scheme, HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT1.ProcessRequestAsync()
2024-07-08 09:50:06.268 -04:00 [INF] Request finished HTTP/1.1 POST htps/psuserver/notificationhub/negotiate?negotiateVersion=1 text/plain;charset=UTF-8 0 - 500 - - 137.7794ms
2024-07-08 09:50:06.268 -04:00 [INF] Request finished HTTP/2 POST htps/psuserver/notificationhub/negotiate?negotiateVersion=1 text/plain;charset=UTF-8 0 - 500 - - 137.8447ms
2024-07-08 09:50:06.268 -04:00 [INF] Request finished HTTP/1.1 POST htps/psuserver/terminalhub/negotiate?negotiateVersion=1 text/plain;charset=UTF-8 0 - 500 - - 137.8455ms
2024-07-08 09:50:07.161 -04:00 [INF] PowerShell Universal performance counters do not exist.
2024-07-08 09:50:09.307 -04:00 [INF] Starting module discovery using Process.
2024-07-08 09:50:10.427 -04:00 [INF] Starting module discovery using Process.
2024-07-08 09:50:10.429 -04:00 [ERR] Error discovering modules
System.ComponentModel.Win32Exception (2): An error occurred trying to start process ‘pwsh’ with working directory ‘C:\ProgramData\UniversalAutomation\Repository’. The system cannot find the file specified.
at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
at UniversalAutomation.JobProcessManager.StartProcessAsCurrentUser(String powerShellPath, String commandLine, Variable credential, Action1 onAgentOutput, Dictionary2 environmentVariables) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Automation\JobProcessManager.cs:line 457
at UniversalAutomation.JobProcessManager.StartModuleDiscovery(ExecutionEnvironment environment) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Automation\JobProcessManager.cs:line 333
at Universal.Server.Services.ModuleService.Discover(ExecutionEnvironment environment) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\ModuleService.cs:line 218
2024-07-08 09:50:10.992 -04:00 [INF] Scheduling script 1 on schedule 0 0 * * * with time zone America/New_York.
2024-07-08 09:50:10.995 -04:00 [INF] Scheduling job ‘update-userproxy’ with schedule update-userproxy on time zone America/New_York
2024-07-08 09:50:11.030 -04:00 [ERR] Failed to schedule job. The time zone ID ‘America/New_York’ was not found on the local computer.
2024-07-08 09:50:11.030 -04:00 [ERR] Failed to read configuration file. schedules.ps1
System.NullReferenceException: Object reference not set to an instance of an object.
at UniversalAutomation.JobService.AddOrUpdateRecurringJob(Script script, Schedule schedule) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Automation\JobService.cs:line 450
at UniversalAutomation.JobService.ScheduleJob(Script script, Schedule schedule) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Automation\JobService.cs:line 401
at UniversalAutomation.Services.SchedulesConfigurationScript.OnReadAfterAddAsync(Schedule schedule) in C:\actions-runner_work\universal\universal\src\Universal.Server\Services\Configuration\Schedules.cs:line 89
at UniversalAutomation.Services.ConfigurationScript`1.ReadAsync(String rootPath, Boolean initialSync) in C:\actions-runner_work\universal\universal\src\PowerShellUniversal\ConfigurationScript.cs:line 284
2024-07-08 09:50:11.194 -04:00 [INF] Running git sync, if configured.
2024-07-08 09:50:11.289 -04:00 [INF] Logging enabled at level Debug
After upgrading to 4.3.2 from 4.2.21 I had the same problem with my scheduled tasks. It couldn’t find my time zone (Europe/Berlin) and I couldn’t find a workaround quickly. Even leaving out the time zones didn’t help.
So I downgraded to 4.2.21 and everything worked again.
I just wanted to wait and see if it would be fixed with the next update.
We bumped the version of our scheduler and they may have a bug in there since this seems to be coming directly from that library. I’ll open an issue with them to see if we can get some more information.
That said, I dug into it a little bit and our custom TimeZoneResolver isn’t being called when scheduling and job and only when attempting to execute the job. This means that it uses the built in TimeZoneInfo class to look up time zones. This can cause issues because Linux doesn’t always have an updated time zone database. Are either of you running on Linux?
I did run the lookup in a PS prompt and it does find Europe/Berlin on Windows and returns the same error when putting in an invalid time zone.
PS C:\Users\adamr> [System.TimeZoneInfo]
IsPublic IsSerial Name BaseType
-------- -------- ---- --------
True True TimeZoneInfo System.Object
PS C:\Users\adamr> [System.TimeZoneInfo]::FindSystemTimeZoneById("Europe/Berlin")
Id : Europe/Berlin
HasIanaId : True
DisplayName : (UTC+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna
StandardName : W. Europe Standard Time
DaylightName : W. Europe Daylight Time
BaseUtcOffset : 01:00:00
SupportsDaylightSavingTime : True
PS C:\Users\adamr> [System.TimeZoneInfo]::FindSystemTimeZoneById("Europe/Berlin222")
MethodInvocationException: Exception calling "FindSystemTimeZoneById" with "1" argument(s): "The time zone ID 'Europe/Berlin222' was not found on the local computer."
PS C:\Users\adamr>
Okay, that’s an interesting behavior.
I have now seen that PowerShell 5.1 is probably the problem.
PowerShell 7 recognizes the timezones as you say, but PowerShell 5 returns an error because it expects a different ID.
PS 5.1:
PS C:\Users> [System.TimeZoneInfo]::FindSystemTimeZoneById("Europe/Berlin")
Ausnahme beim Aufrufen von "FindSystemTimeZoneById" mit 1 Argument(en): "Die Zeitzonen-ID "Europe/Berlin" wurde auf
dem lokalen Computer nicht gefunden."
In Zeile:1 Zeichen:1
+ [System.TimeZoneInfo]::FindSystemTimeZoneById("Europe/Berlin")
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : TimeZoneNotFoundException
But wih another ID:
PS C:\Users> [System.TimeZoneInfo]::FindSystemTimeZoneById("W. Europe Standard Time")
Id : W. Europe Standard Time
DisplayName : (UTC+01:00) Amsterdam, Berlin, Bern, Rom, Stockholm, Wien
StandardName : Mitteleuropäische Zeit
DaylightName : Mitteleuropäische Sommerzeit
BaseUtcOffset : 01:00:00
SupportsDaylightSavingTime : True
And 5.1 is still my primary environment on the server, so that seems to be where the error is coming from.
I’m on windows server 2016. I am also using the 5.1 environment but the scheduler was mostly working before on the older version of PSU so I think 5.1 itself should not be the problem? Unless there was some other backend change with PSU.
This call is being made within PSU and not in any PowerShell environment. It would be more similar to the PS7 environment. Aside from the upgraded Hangfire package, we haven’t changed this particular time zone lookup in years so I have to imagine it’s coming from that.
I’ll open an issue in our tracker. We may be able to just revert the hangfire update but I can’t reproduce this issue so it’s hard to say if that’s actually the solution.
It does look like the timezone id is eastern standard time when I run it in the integrated environment.
PS C:\ProgramData\UniversalAutomation\Repository> get-timezone
Id : Eastern Standard Time
HasIanaId : False
DisplayName : (UTC-05:00) Eastern Time (US & Canada)
StandardName : Eastern Standard Time
DaylightName : Eastern Daylight Time
BaseUtcOffset : -05:00:00
SupportsDaylightSavingTime : True
I changed the timezone on the schedule for the script to UTC and it looks like it works albeit on UTC time. I saw there was also a US/Eastern timezone but that doesn’t work either.
I am running into an issue where the scheduled jobs only run sometimes? I see in the job log that the jobs successfully ran because of schedule but it is supposed to run everyday and it looks like it only runs some days. Like the last ran scheduled jobs are from 3 days ago. I’m not sure why it runs only sometimes.