Universal.Server process high CPU and memory usage

PowerShell Universal
1 
Product: PowerShell Universal
Version: 2.7.3

Hello, I just updated 2.3.2 to 2.7.3 and dotnet-hosting-5.0.13-win package running on IIS 10 and 2016 Standard .
When users login to the dashboards Universal.Server and the associated Powershell processes start spiking the CPU up to 99%, eventually getting 502 Bad Gateway errors.
Also the Universal.Server process has a very high memory usage.
With log debugging turned on I can see Socket errors?

image
image833×314 59.8 KB

Microsoft.AspNetCore.Connections.ConnectionResetException: An existing connection was forcibly closed by the remote host.
—> System.Net.Sockets.SocketException (10054): An existing connection was forcibly closed by the remote host.
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.g__ThrowSocketException|7_0(SocketError e)
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketAwaitableEventArgs.GetResult()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.ProcessReceives()
at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.Internal.SocketConnection.DoReceive()
— End of inner exception stack trace —
at System.IO.Pipelines.PipeCompletion.ThrowLatchedException()
at System.IO.Pipelines.Pipe.GetReadResult(ReadResult& result)
at System.IO.Pipelines.Pipe.ReadAsync(CancellationToken token)
at System.IO.Pipelines.Pipe.DefaultPipeReader.ReadAsync(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1ContentLengthMessageBody.ReadAsyncInternal(CancellationToken cancellationToken)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpRequestStream.ReadAsyncInternal(Memory1 destination, CancellationToken cancellationToken) at System.IO.StreamReader.ReadBufferAsync(CancellationToken cancellationToken) at System.IO.StreamReader.ReadToEndAsyncInternal() at UniversalDashboard.Controllers.UDComponentController.ElementPost(String id, Boolean form) in D:\a\universal\universal\src\Universal.Server\Controllers\UDComponentController.cs:line 377 at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 134
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Universal.Server.Middleware.SwaggerAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\universal\universal\src\Universal.Server\Middleware\SwaggerAuthMiddleware.cs:line 42
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<b__1>d.MoveNext()
— End of stack trace from previous location —
at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
2022-02-02 15:58:06.105 -05:00 [WRN] No exception handler was found, rethrowing original exception.
2022-02-02 15:58:06.105 -05:00 [ERR] Connection id “0HMF6GQR7QFJD”, Request id “0HMF6GQR7QFJD:00000002”: An unhandled exception was thrown by the application

1 Like
2

We’ve been actively looking into high memory usage issues in PSU. The latest 2.8 build has a lot of improvements to memory usage that might remedy this. You can download it here. Make sure that you tried 2.8 and not 3.0.

We have also added a new option to environments to recycle runspaces. If you add the -RunspaceRecycling parameter onto New-PSUEnvironment in environments.ps1, it will clean up runspaces after a certain amount time or usages.

3

Thank you for the quick reply, I installed the update 2.8.0 this morning and added the -RunspaceRecycling at first it worked low memory and CPU.
But then I got this error in the Application logs and CPU and memory spiked again for the Universal.Server process.

Application: Universal.Server.exe
CoreCLR Version: 5.0.921.35908
.NET Version: 5.0.9
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AggregateException: One or more hosted services failed to stop. (The operation was canceled.)
—> System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at Hangfire.Processing.TaskExtensions.WaitOneAsync(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token)
at Hangfire.Processing.BackgroundDispatcher.WaitAsync(TimeSpan timeout, CancellationToken cancellationToken)
at Hangfire.Server.BackgroundProcessingServer.WaitForShutdownAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
— End of inner exception stack trace —
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
at Universal.Server.Program.<>c__DisplayClass3_0.b__0(Options o) in D:\a\universal\universal\src\Universal.Server\Program.cs:line 63
at CommandLine.ParserResultExtensions.WithParsed[T](ParserResult1 result, Action1 action)
at Universal.Server.Program.Main(String[] args) in D:\a\universal\universal\src\Universal.Server\Program.cs:line 45

#########################################

Faulting application name: Universal.Server.exe, version: 1.0.0.0, time stamp: 0x60e896d9
Faulting module name: KERNELBASE.dll, version: 10.0.14393.4770, time stamp: 0x6180883c
Exception code: 0xe0434352
Fault offset: 0x0000000000034f38
Faulting process id: 0x5ba0
Faulting application start time: 0x01d818f4937ab24f
Faulting application path: F:\inetpub\Universal\Universal.Server.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: ee4aa8df-98ed-44da-8aea-8e4d0bbc0b41
Faulting package full name:
Faulting package-relative application ID:

4

Can you tell me a little more about your configuration?

I’m looking for default environment, authentication type, number of roles etc.

5

Default Environment is right now PWSH 7.2.1 I also tried 5.1 same result.
Authentication I am using the PSUAuthenticationMethod with a custom script.
4 custom roles with a AD lookup script
plus Administrator, Reader and Operator no scripting.

6

For the AD lookup script, are you using the ActiveDirectory module? Can you let me know which version?

Can you also let me know what you have set for Security Environment?

image
image1062×510 7.54 KB

7

One more tidbit that I think we need to investigate. Can you please click details in task manager and show the command line for each of those PowerShell processes?

image
image1134×157 22.9 KB

I’m just wondering if PowerShell 7 is doing implicit module compat loading. I can force this to happen by using Import-Module TestModule -UseWindowsPowerShell but PS7 will also do this automatically if you don’t disable it.

The reason I’m asking about the WinPS processes are those may be the Windows Compat sessions loading up. What’s troublesome about that is that there will be a process per runspace in PowerShell Universal.

8

Yes I am using the AD module ver. 1.0
I misunderstood default environment I was thinking of the individual dashboards this shows the integrated environment.

image
image2103×536 54.1 KB

Here are the PoS processes

Get-Process | where {$.name -eq “pwsh” -or $.name -eq “powershell”} | Select-Object { $_.commandline } | fl

$_.commandline : “C:\Program Files\PowerShell\7\pwsh.exe” -NoProfile -Command “& {
[System.Reflection.Assembly]::LoadFrom(‘F:\inetpub\Universal\Host\host.dll’) | Out-Null;
[UniversalHost.AgentService]::StartDashboard(64082, 11204, 51249) }”

$_.commandline : “C:\Program Files\PowerShell\7\pwsh.exe” -NoProfile -Command “& {
[System.Reflection.Assembly]::LoadFrom(‘F:\inetpub\Universal\Host\host.dll’) | Out-Null;
[UniversalHost.AgentService]::StartDashboard(64068, 11204, 51249) }”

$_.commandline : “C:\Program Files\PowerShell\7\pwsh.exe” -NoProfile -Command “& {
[System.Reflection.Assembly]::LoadFrom(‘F:\inetpub\Universal\Host\host.dll’) | Out-Null;
[UniversalHost.AgentService]::StartDashboard(64055, 11204, 51249) }”

$_.commandline : “C:\Program Files\PowerShell\7\pwsh.exe” -NoProfile -Command “& {
[System.Reflection.Assembly]::LoadFrom(‘F:\inetpub\Universal\Host\host.dll’) | Out-Null;
[UniversalHost.AgentService]::StartDashboard(64077, 11204, 51249) }”

$_.commandline : “C:\Program Files\PowerShell\7\pwsh.exe” -NoProfile -Command “& {
[System.Reflection.Assembly]::LoadFrom(‘F:\inetpub\Universal\Host\host.dll’) | Out-Null;
[UniversalHost.AgentService]::StartDashboard(64038, 11204, 51249) }”

$_.commandline : “C:\Program Files\PowerShell\7\pwsh.exe”

$_.commandline : “C:\Program Files\PowerShell\7\pwsh.exe” -NoProfile -Command “& {
[System.Reflection.Assembly]::LoadFrom(‘F:\inetpub\Universal\Host\host.dll’) | Out-Null;
[UniversalHost.AgentService]::StartDashboard(64090, 11204, 51249) }”

9

Can we schedule a call to look at this together? I’d like to do some additional debugging.

10

That sounds good, I am open all afternoon EST do you wan to send an invite Or should I send one (MsTeams).

Thank you,

Reiner

11

If anyone is having a similar issue to this, it seems that the latest 2.8 nightly build has helped to resolve out of control runspace creation and high CPU\memory usage.

1 Like
12

image
image1281×471 71.7 KB

We are seeing this on 4.2.12.
This will eventually consume all memory.
My powershell 5.x environments are not doing this.