More v4 to v5 upgrade problems. :)

I am unable to run any scripts in the 5.0.5 update. It runs then times out after the timeout threshold and gives and error that powershell could not respond in a timely fashion.
Here is the log I found when trying to run a script:

2024-09-06 00:00:07.188 -04:00 [WRN] Failed to process the job '465946e9-3b14-4905-8ee8-600082c8c18e': an exception occured. Job was automatically deleted because the retry attempt count exceeded 0.
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at PowerShellUniversal.Extensibility.EFTable`2.Update(T item) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 282
   at UniversalAutomation.HeartBeatService.Run()
   at InvokeStub_HeartBeatService.Run(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
2024-09-06 00:00:10.502 -04:00 [INF] PSModulePath (PowerShell 7) - C:\inetpub\wwwroot\Agents\5.1\Modules;C:\Windows\system32\config\systemprofile\Documents\PowerShell\Modules;C:\Program Files\PowerShell\Modules;C:\inetpub\wwwroot\Modules;C:\inetpub\wwwroot\Modules;C:\Program Files\Microsoft Azure AD Connect Health Agent\Modules\;C:\ProgramData\UniversalAutomation\Repository\Modules;C:\inetpub\wwwroot\UniversalDashboard\Components;C:\ProgramData\UniversalAutomation\Repository\Components
2024-09-06 00:00:11.265 -04:00 [INF] PSModulePath (PowerShell 7) - C:\inetpub\wwwroot\Modules;C:\Windows\system32\config\systemprofile\Documents\PowerShell\Modules;C:\Program Files\PowerShell\Modules;C:\inetpub\wwwroot\Modules;C:\inetpub\wwwroot\Modules;C:\Program Files\Microsoft Azure AD Connect Health Agent\Modules\;C:\ProgramData\UniversalAutomation\Repository\Modules;C:\inetpub\wwwroot\UniversalDashboard\Components;C:\ProgramData\UniversalAutomation\Repository\Components
2024-09-06 00:02:07.201 -04:00 [WRN] Failed to process the job 'e3a168ca-fa74-45de-ac8d-2b67e927d84f': an exception occured. Job was automatically deleted because the retry attempt count exceeded 0.
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at PowerShellUniversal.Extensibility.EFTable`2.Update(T item) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 282
   at UniversalAutomation.HeartBeatService.Run()
   at InvokeStub_HeartBeatService.Run(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
2024-09-06 00:04:07.217 -04:00 [WRN] Failed to process the job 'e61d05fe-1dd7-45cd-adef-4030e851a766': an exception occured. Job was automatically deleted because the retry attempt count exceeded 0.
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at PowerShellUniversal.Extensibility.EFTable`2.Update(T item) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 282
   at UniversalAutomation.HeartBeatService.Run()
   at InvokeStub_HeartBeatService.Run(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
2024-09-06 00:05:07.230 -04:00 [WRN] Failed to process the job '235d68bb-335b-42a3-8dc4-233dc6eb040e': an exception occured. Job was automatically deleted because the retry attempt count exceeded 0.
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at PowerShellUniversal.Extensibility.EFTable`2.Update(T item) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 282
   at UniversalAutomation.HeartBeatService.Run()
   at InvokeStub_HeartBeatService.Run(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
2024-09-06 00:08:07.255 -04:00 [WRN] Failed to process the job 'c5fed3f2-3397-4972-981f-a68035980a05': an exception occured. Job was automatically deleted because the retry attempt count exceeded 0.
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at PowerShellUniversal.Extensibility.EFTable`2.Update(T item) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 282
   at UniversalAutomation.HeartBeatService.Run()
   at InvokeStub_HeartBeatService.Run(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
2024-09-06 00:09:07.263 -04:00 [WRN] Failed to process the job '858a25a5-6504-468d-a8f6-2e5d0e4ec420': an exception occured. Job was automatically deleted because the retry attempt count exceeded 0.
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 5: 'database is locked'.
   at PowerShellUniversal.Extensibility.EFTable`2.Update(T item) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 282
   at UniversalAutomation.HeartBeatService.Run()
   at InvokeStub_HeartBeatService.Run(Object, Object, IntPtr*)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

This continues on the for the timeout period.

The next issue I have is the scheduler can not find the US/Eastern time zone again (this happened in v4 as well but was patched). So I have set my schedules to UTC for now.

Product: PowerShell Universal
Version: 1.4.6

Looks like your DB is locked. Did you change the password to the DB during the upgrade?

If you’re using a custom appsettings.json to change some of the default behaviors, you may have the same configuration I had during my upgrade, which had to be changed in order for PSU to be able to access the DB.

The following line needed to be changed:

What it was in v4:

"Data": {
	"ConnectionString": "filename=%ProgramData%\\UniversalAutomation\\database.db"
}

What it needs to be in v5:

"Data": {
	"ConnectionString": "Data Source=%ProgramData%\\UniversalAutomation\\database.db"
}

I think this is resolved in the nightly build.

https://imsreleases.blob.core.windows.net/universal-nightly/10730058765/release.md

  • Fixed a SQLite locking issue when running many jobs at once
1 Like

OK, I updated to the latest nightly and no db locks but scripts still time out. Here is the log:

2024-09-06 11:19:00.935 -04:00 [INF] Error executing job 8: PowerShell did not respond in a timely fashion. PowerShell Universal is currently waiting 30 seconds for PowerShell to start running the script.
You can increase the timeout by setting the 'Job Handshake Timeout' setting in the Settings \ General \ Automation page.
If you are still experiencing issues, you may need to consider increasing the resources available on your system or take advantage of the Random Delay parameter for schedules to stagger jobs.
    at UniversalAutomation.Services.GrpcExecutionService.ExecuteScript(Job job, ExecutionEnvironment environment, ExecuteScriptRequest request, ExecutionCallback executionCallback) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\GrpcExecutionService.cs:line 113
   at UniversalAutomation.ExecutionService.ExecutePowerShell(Script script, IEnumerable`1 parameters, Job job, Schedule schedule, ExecutionCallback callback) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 732
   at UniversalAutomation.ExecutionService.TryExecute(Job job, Schedule schedule) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 344
   at UniversalAutomation.ExecutionService.TryExecute(Job job, Schedule schedule) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 436
   at UniversalAutomation.ExecutionService.Execute(Int64 jobId, Nullable`1 scheduleId) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\Automation\ExecutionService.cs:line 161 System.Exception
2024-09-06 11:19:18.126 -04:00 [INF] Starting heartbeat job.
2024-09-06 11:19:18.151 -04:00 [INF] Finished heartbeat job.
2024-09-06 11:19:30.241 -04:00 [ERR] Navigation failed when changing the location to ./admin/automation/jobs/8
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.JSInterop.JSRuntime.InvokeAsync[TValue](Int64 targetInstanceId, String identifier, Object[] args)
   at Microsoft.JSInterop.JSRuntimeExtensions.InvokeVoidAsync(IJSRuntime jsRuntime, String identifier, Object[] args)
   at Microsoft.AspNetCore.Components.Server.Circuits.RemoteNavigationManager.<>c__DisplayClass13_0.<<NavigateToCore>g__PerformNavigationAsync|0>d.MoveNext()
2024-09-06 11:19:30.246 -04:00 [ERR] Unhandled exception in circuit 'bNM3eQWh2DOzpi7p21RP6vubKCPlSR64hWe5Oom7ht4'.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.JSInterop.JSRuntime.InvokeAsync[TValue](Int64 targetInstanceId, String identifier, Object[] args)
   at Microsoft.JSInterop.JSRuntimeExtensions.InvokeVoidAsync(IJSRuntime jsRuntime, String identifier, Object[] args)
   at Microsoft.AspNetCore.Components.Server.Circuits.RemoteNavigationManager.<>c__DisplayClass13_0.<<NavigateToCore>g__PerformNavigationAsync|0>d.MoveNext()

Are you using the PowerShell 7 environment or something else? Can you share environments.ps1, if it exists?

I don’t see an environment.ps1 in the main folder or under the C:\ProgramData\UniversalAutomation folder. I am using powershell 5.1 environment though.

When I go to terminals I can access one that is running the integrated environment but when I try to access a terminal with powershell 5.1 I get this

An error occured. Failed to connect to terminal. Process exited.
at PowerShellUniversal.Services.Terminals.TerminalManager.StartAsync(Terminal terminal, Identity identity) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\TerminalManager.cs:line 66 at PowerShellUniversal.StartTerminalButton.OpenTerminal() in C:\actions-runner\_work\universal\universal\src\Universal.Server\Shared\Automation\StartTerminalButton.razor:line 19 at Microsoft.AspNetCore.Components.ComponentBase.CallStateHasChangedOnAsyncCompletion(Task task) at AntDesign.Button.HandleOnClick(MouseEventArgs args) at Microsoft.AspNetCore.Components.ComponentBase.CallStateHasChangedOnAsyncCompletion(Task task) at Microsoft.AspNetCore.Components.RenderTree.Renderer.GetErrorHandledTask(Task taskToHandle, ComponentState owningComponentState)

Here is the log file

2024-09-06 13:52:53.019 -04:00 [INF] Starting job using Process as SYSTEM.
2024-09-06 13:52:54.172 -04:00 [WRN] Unhandled exception rendering component: Failed to connect to terminal. Process exited.
System.Exception: Failed to connect to terminal. Process exited.
   at PowerShellUniversal.Services.Terminals.TerminalManager.StartAsync(Terminal terminal, Identity identity) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\TerminalManager.cs:line 66
   at PowerShellUniversal.StartTerminalButton.OpenTerminal() in C:\actions-runner\_work\universal\universal\src\Universal.Server\Shared\Automation\StartTerminalButton.razor:line 19
   at Microsoft.AspNetCore.Components.ComponentBase.CallStateHasChangedOnAsyncCompletion(Task task)
   at AntDesign.Button.HandleOnClick(MouseEventArgs args)
   at Microsoft.AspNetCore.Components.ComponentBase.CallStateHasChangedOnAsyncCompletion(Task task)
   at Microsoft.AspNetCore.Components.RenderTree.Renderer.GetErrorHandledTask(Task taskToHandle, ComponentState owningComponentState)
2024-09-06 13:53:05.518 -04:00 [INF] Starting groom job.
2024-09-06 13:53:05.520 -04:00 [INF] Script Update-userproxy.ps1 has a max history of 100
2024-09-06 13:53:05.523 -04:00 [INF] Script has 0 jobs to delete.
2024-09-06 13:53:05.523 -04:00 [INF] Script rebootpending-win10.ps1 has a max history of 100
2024-09-06 13:53:05.525 -04:00 [INF] Script has 0 jobs to delete.
2024-09-06 13:53:05.525 -04:00 [INF] Script cleanoldadcomp.ps1 has a max history of 100
2024-09-06 13:53:05.527 -04:00 [INF] Script has 0 jobs to delete.
2024-09-06 13:53:05.527 -04:00 [INF] Script disable-nopwreq.ps1 has a max history of 100
2024-09-06 13:53:05.529 -04:00 [INF] Script has 0 jobs to delete.
2024-09-06 13:53:05.529 -04:00 [INF] Script Delete-oldaduser.ps1 has a max history of 100
2024-09-06 13:53:05.530 -04:00 [INF] Script has 0 jobs to delete.
2024-09-06 13:53:05.530 -04:00 [INF] Script VaronisServices.ps1 has a max history of 100
2024-09-06 13:53:05.532 -04:00 [INF] Script has 0 jobs to delete.
2024-09-06 13:53:05.533 -04:00 [INF] Groom date is: 8/7/2024 5:53:05 PM
2024-09-06 13:53:05.549 -04:00 [ERR] Failed to groom.
System.InvalidOperationException: The LINQ expression 'DbSet<Terminal>()
    .Where(t => t.Name
        .EqualsIgnoreCase(__ti_Terminal_0))' could not be translated. Additional information: Translation of method 'PowerShellUniversal.StringExtensions.EqualsIgnoreCase' failed. If this method can be mapped to your custom function, see https://go.microsoft.com/fwlink/?linkid=2132413 for more information. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.RelationalQueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Storage.Database.CompileQuery[TResult](Expression query, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.CompileQueryCore[TResult](IDatabase database, Expression query, IModel model, Boolean async)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   at PowerShellUniversal.MemoryTable`1..ctor(IEnumerable`1 items) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal\MemoryTable.cs:line 10
   at PowerShellUniversal.Extensibility.EFTable`2.Where(Expression`1 predicate) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 483
   at UniversalAutomation.GroomService.GroomIdleTerminals() in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\GroomService.cs:line 52
   at UniversalAutomation.GroomService.Groom() in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\GroomService.cs:line 308
2024-09-06 13:53:05.579 -04:00 [INF] Finished groom job.

I tried updating to 5.0.6 and still same problem.

I’ve opened an issue to track this. Still trying to reproduce myself.

1 Like

Do you want me to do any debugging? Our PSU is kind of dead in the water at the moment and I’m just manually running daily scripts.

1 Like

We have changes for WinPS in v5.0.7 to correct a problem with the PSModulePath environment variable that may be the culprit. If you get the latest nightly build, it has this change if you want to see if that resolves it for you as well.

OK, I tried 5.0.7 nightly and still same errors.

Can you please open a support case?

Support is saying the issue is with my environment and outside of PSU scope lol. Maybe I’ll set up a linux vm and try to load it on there.