.NET Errors on API Calls

1

Hello @adam,

Have you ever experienced this kind of .NET errors ?

Category: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware
EventId: 1
ConnectionId: 0HM29Q17DJBGR
RequestId: 0HM29Q17DJBGR:00000001
RequestPath: /v2/vault
SpanId: a027a7fdcd9a8a4e
TraceId: 49f06b1751394243b352117de8a41934
ParentId: 0000000000000000

An unhandled exception has occurred while executing the request.

Exception:
Grpc.Core.RpcException: Status(StatusCode=Cancelled, Detail=“Cancelled”)
at Universal.Server.Services.ApiProxy.ExecuteAsync(HttpContext httpContext) in D:\a\universal\universal\src\Universal.Server\Services\ApiProxy.cs:line 186
at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 58
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

I have daily around 2.5k clients sending status to the API endpoint around 11h15 and everyday the server gets overwelm with lsass.exe taking all the CPU and errors like that flooding the Windows / application logs.

Any help on this would be greatly appreciated.

had the same experience with 1.3.0 and just updated to 1.3.1 to test if it would get better and still the same.

PSU is running on a Windows Server 2019 (1809) with IIS 10. HTTPS is enabled IIS and on Kestrel

2

Could this be in relation with the default threadpool settings that could lead to a thread pool starvation ?

3

I haven’t seen this before. Are you using Windows Auth? That might be why lsass.exe is going crazy. Not exactly sure what cancelled means but I have a hunch something is starved for sure.

Are you sending 2.5k requests or do you have 2.5k clients sending many requests?

4

Yep I’m using Windows Authentication with IIS.

it’s ~2.5k clients sending POST request to an Endpoint everyday @ 11h15 and lsass.exe goes crazy at 100% CPU use on all 8 cores of the VM running PSU.

I followed this MS post and updated the value and I’ll check today if this “starvation” issue still coming.

I’ve read somewhere that it may come from the fact that the cert I’m using could be 4096 and changing it with a 2048 can help with that lsass issue.

I’ll do some tests and investigation and I’ll keep you posted. Let me know if you find anything on your end !

Many thanks in advance for your help and assistance