PowerShell Universal - 5.1.0

PowerShell Universal releases
21

Groom is failing here after the update. Any ideas?

[ERR][UniversalAutomation.GroomService] Failed to groom.
System.InvalidOperationException: The LINQ expression 'DbSet<Terminal>()
    .Where(t => t.Name
        .EqualsIgnoreCase(__ti_Terminal_0))' could not be translated. Additional information: Translation of method 'PowerShellUniversal.StringExtensions.EqualsIgnoreCase' failed. If this method can be mapped to your custom function, see https://go.microsoft.com/fwlink/?linkid=2132413 for more information. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to 'AsEnumerable', 'AsAsyncEnumerable', 'ToList', or 'ToListAsync'. See https://go.microsoft.com/fwlink/?linkid=2101038 for more information.
   at Microsoft.EntityFrameworkCore.Query.QueryableMethodTranslatingExpressionVisitor.Translate(Expression expression)
   at Microsoft.EntityFrameworkCore.Query.QueryCompilationContext.CreateQueryExecutor[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass9_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Query.Internal.CompiledQueryCache.GetOrAddQuery[TResult](Object cacheKey, Func`1 compiler)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.Execute[TResult](Expression query)
   at Microsoft.EntityFrameworkCore.Query.Internal.EntityQueryable`1.GetEnumerator()
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)
   at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector)
   at PowerShellUniversal.MemoryTable`1..ctor(IEnumerable`1 items) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal\MemoryTable.cs:line 10
   at PowerShellUniversal.Extensibility.EFTable`2.Where(Expression`1 predicate) in C:\actions-runner\_work\universal\universal\src\PowerShellUniversal.Extensibility\Persistence\EFTable.cs:line 570
   at UniversalAutomation.GroomService.GroomIdleTerminals() in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\GroomService.cs:line 54
   at UniversalAutomation.GroomService.Groom() in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\GroomService.cs:line 296
22

Can you let me know what DB type you are using?

23

SQLite. It was an upgrade from 4.4.1 if that makes a difference.

1 Like
24

I just restored a snapshot to roll back to 5.0.16. 5.1.0 was causing a memory leak. The Universal.Server.exe was taking 4.9 GB of memory, and the entire server was unresponsive. No changes to any scripts or schedules were made post-upgrade from 5.0.16 to 5.1.0.

25

Seeing the same here, after 24hr or so Iā€™m up to ~5gb. Only 2 apps, normal usage seems like 1gb after a restart.

I do update a large hashtable every hour into the persistent cache, I was wondering if it wasnā€™t purging the old values automatically when overwritten.

26

If would be very helpful if either of you could collect a memory dump that I can analyze. I can provide a dropbox link in a DM to upload it to.

2 Likes
27

Iā€™ve already gone back to 5.0.16 so I canā€™t provide that unless I upgrade again. I will upgrade my test environment. but we donā€™t run anything there, so I donā€™t know for sure that itā€™ll happen on that system.

1 Like
28

5.1.0 upgrade went well for me.
4 days up, I just checked memory and we are at 2GB, but the server is not being used heavily.
I have process explorer running and keep an eye on it.

29

I get the same error when editing apps - I also get it when editing endpoints and roles.
Editing scripts doesnā€™t give me the error.

image
image1895Ɨ210 22.8 KB

Also, unable to view root directory of scripts in folder view, but can see them in list view. It is only the root folder (Scripts folder in the image) if I select a sub folder the scripts within that folder are visible.

image
image1886Ɨ650 32.4 KB

image
image1899Ɨ691 44.2 KB

MSI Install
SQL Database
Upgrade from 5.0.16

30

So far my dev. environment hasnā€™t had a memory issue, but itā€™s literally not doing anything other than running whatever internal healthchecks and tasks PSU runs on any installation.

31

I think weā€™ve found root cause for the memory issue. Entity Framework (our DB ORM), it eagerly loading tons of data during job executions. This is causing both large amounts of data to be returned from the database and then equally amounts of large data to be updated in the database. This puts a ton of strain on both PSU and the DB. Additionally, the eager loading is happening when using Get-PSUJob or viewing the jobs table, which can put more load on the system.

On small, less busy environments it was likely not noticeable but once that jobs table reaches a certain size, it will be. It is possible to reproduce this by running 10s of jobs at once or by using child jobs. It will also cause jobs to have invalid state and time stamps. It looks like this was a problem since somewhere in the 5.0.x release but Iā€™m not clear on the specific version or why 5.1 exasperated the problem.

Weā€™ll have a 5.1.2 release coming out shortly to address this.

3 Likes
32

Oh. Well, that sounds likely. We do have some very heavy hitting jobs that run overnight, and some do do use Get-PSUJob.