Hello,
Recently, we had an issue with our two node setup. Jobs were getting sent to a node, and getting stuck in queue, which is being discussed in another post. Jobs stuck in queue - PowerShell Universal - Ironman Software Forums
Typically, we will clear these jobs using SQL, then reboot the PSU servers to ‘refresh’ everything. However, upon reboot, one of our nodes suddenly began failing to start. Investigating into the issue, we were receiving an HTTP 500 error and the following line repeated in application logs (with job number changing).
[WRN] Failed to process the job '10294269': an exception occured. Job was automatically deleted because the retry attempt count exceeded 0.
System.InvalidOperationException: Timeout expired. The timeout period elapsed prior to obtaining a connection from the pool. This may have occurred because all pooled connections were in use and max pool size was reached.
at Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection(DbConnection owningConnection, TaskCompletionSource`1 retry, DbConnectionOptions userOptions, DbConnectionInternal oldConnection, DbConnectionInternal& connection)
at Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal(DbConnection outerConnection, DbConnectionFactory connectionFactory, TaskCompletionSource`1 retry, DbConnectionOptions userOptions)
at Microsoft.Data.SqlClient.SqlConnection.TryOpen(TaskCompletionSource`1 retry, SqlConnectionOverrides overrides)
at Microsoft.Data.SqlClient.SqlConnection.Open(SqlConnectionOverrides overrides)
at Microsoft.EntityFrameworkCore.SqlServer.Storage.Internal.SqlServerConnection.OpenDbConnection(Boolean errorsExpected)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenInternal(Boolean errorsExpected)
at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.Execute(IEnumerable`1 commandBatches, IRelationalConnection connection)
at Microsoft.EntityFrameworkCore.Storage.RelationalDatabase.SaveChanges(IList`1 entries)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(IList`1 entriesToSave)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(StateManager stateManager, Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.<>c.<SaveChanges>b__107_0(DbContext _, ValueTuple`2 t)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.<>c__DisplayClass28_0`2.<Execute>b__0(DbContext context, TState state)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.ExecuteImplementation[TState,TResult](Func`3 operation, Func`3 verifySucceeded, TState state)
at Microsoft.EntityFrameworkCore.Storage.ExecutionStrategy.Execute[TState,TResult](TState state, Func`3 operation, Func`3 verifySucceeded)
at Microsoft.EntityFrameworkCore.ChangeTracking.Internal.StateManager.SaveChanges(Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.DbContext.SaveChanges(Boolean acceptAllChangesOnSuccess)
at Microsoft.EntityFrameworkCore.DbContext.SaveChanges()
at PowerShellUniversal.SQL.SqlTable`1.Update(T item) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.SQL\Database.cs:line 659
at UniversalAutomation.HeartBeatService.Run() in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\HeartBeatService.cs:line 78
at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
2025-01-30 12:10:40.199 -06:00 [WRN] Slow log: (Hangfire.DisableConcurrentExecutionAttribute, Hangfire.Server.PerformingContext) performed "OnPerforming for 10295623" in 90 sec
Additionally, when trying to actually hit the web page, the following log is shown
2025-01-30 12:15:03.218 -06:00 [ERR] Connection id "0HNA1B4VJNUJK", Request id "0HNA1B4VJNUJK:00000007": An unhandled exception was thrown by the application.
System.InvalidOperationException: Provide Authority, MetadataAddress, Configuration, or ConfigurationManager to OpenIdConnectOptions
at Microsoft.AspNetCore.Authentication.OpenIdConnect.OpenIdConnectOptions.Validate()
at Microsoft.AspNetCore.Authentication.RemoteAuthenticationOptions.Validate(String scheme)
at Microsoft.AspNetCore.Authentication.AuthenticationBuilder.<>c__DisplayClass4_0`2.<AddSchemeHelper>b__1(TOptions o)
at Microsoft.Extensions.Options.ValidateOptions`1.Validate(String name, TOptions options)
at Microsoft.Extensions.Options.OptionsFactory`1.Create(String name)
at Microsoft.Extensions.Options.OptionsMonitor`1.<>c.<Get>b__10_0(String name, IOptionsFactory`1 factory)
at Microsoft.Extensions.Options.OptionsCache`1.<>c__DisplayClass3_1`1.<GetOrAdd>b__2()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)
--- End of stack trace from previous location ---
at System.Lazy`1.CreateValue()
at Microsoft.Extensions.Options.OptionsCache`1.GetOrAdd[TArg](String name, Func`3 createOptions, TArg factoryArgument)
at Microsoft.Extensions.Options.OptionsMonitor`1.Get(String name)
at Microsoft.AspNetCore.Authentication.AuthenticationHandler`1.InitializeAsync(AuthenticationScheme scheme, HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationHandlerProvider.GetHandlerAsync(HttpContext context, String authenticationScheme)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
I’ve attempted a re-install, as well as version upgrade. At this point, short of wiping the OS, I am unsure how to proceed. Has anyone experienced similar issues? Of note, we’ve been using this setup since early 3.X. So I am questioning if the SQL DBs have just undergone so many upgrades that its starting to fail.
Product: PowerShell Universal
Version: 4.3.4