PSU Service Crashing

PowerShell Universal
1 
Product: PowerShell Universal
Version: 1.5.15

I’ve noticed that the PSU service has crashed a few per hour. Is there something other than the event viewer I can look at to see if it’s code causing this or if something else is triggering it?

All the events for crashes have the same data:

EventId: 1
ConnectionId: 0HM953J3BRFLT
RequestId: 0HM953J3BRFLT:00000001
RequestPath: /register
SpanId: 3e54c9d494cf4340
TraceId: 673c640d95b3a2429a9900cd1e8d4e96
ParentId: 0000000000000000

An unhandled exception has occurred while executing the request.

Exception: 
Grpc.Core.RpcException: Status(StatusCode=Cancelled, Detail="Cancelled")
   at Universal.Server.Services.ApiProxy.ExecuteAsync(HttpContext httpContext) in D:\a\universal\universal\src\Universal.Server\Services\ApiProxy.cs:line 327
   at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 61
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
type or paste code here
2

I think it may be the load on the server itself. I looked up some of the errors and it looks like rate limiting is kicking in.

I found the rate limiting package in GitHub, but I have no idea how to change the limits on PSU
GitHub - stefanprodan/AspNetCoreRateLimit: ASP.NET Core rate limiting middleware.

3

The rate limiting is only listed in the stack trace because we add the middleware. If you haven’t configured rate limiting, then that is not coming into effect.

As for the crash, can you enable debug logging in the appsettings.json file and try to reproduce again? Does the entire server crash? It’s a bit surprising since this is in a request which should be handled by ASP.NET.

You can also enable crash dumps to see if a DMP file is generated: How to generate crash dumps on a Windows machine

4

@adam the OS stays running without issues. The dashboards still load and respond fine, endpoints on the other hand stop responding until the universal service is restarted. I made the change to logging and caught the “crash”.

5

Ah ok. Yep. The API process is terminating for some reason. Can you look in the Windows event application log to see if there are any .NET errors related to PowerShell.exe?

I will enhance this at least to restart the API process in this circumstance so restarting the service manually isn’t necessary. It will at least provide some resilience in a case such as this.

6

I have lots of application event ID 1 with the source being Universal.Server but nothing from powershell.exe

image

7

@adam is there any other place where I can take a look at logging? I would like to try an fix the underlying issue itself.

I’m a paid user so if we need to use up one of my 1-on-1 email support slots I’d be use one up to do so.

Thank you.

8

Let’s schedule something. Can you email me at support@ironmansoftware.com? I’d like to take a closer look.

9

Will do, thanks.