Docker API issue - Socket closed/failed to connect to all addresses

PowerShell Universal Universal API
1

Hi there,

I’m running into issues reaching any API endpoints with the powershell universal docker image (currently using 1.5.19-ubunutu-18.04 )
The web frontend is still available, however all API endpoints return empty with an HTTP status code of zero - similar to the issues mentioned in passing previously

I see the below logs on the containers console

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
Grpc.Core.RpcException: Status(StatusCode=Unavailable, Detail=“Socket closed”)
at Universal.Server.Services.ApiProxy.ExecuteAsync(HttpContext httpContext) in D:\a\universal\universal\src\Universal.Server\Services\ApiProxy.cs:line 327
at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 61
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at AspNetCoreRateLimit.RateLimitMiddleware1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109 at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1] An unhandled exception has occurred while executing the request. Grpc.Core.RpcException: Status(StatusCode=Unavailable, Detail="failed to connect to all addresses") at Universal.Server.Services.ApiProxy.ExecuteAsync(HttpContext httpContext) in D:\a\universal\universal\src\Universal.Server\Services\ApiProxy.cs:line 357 at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 61 at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at AspNetCoreRateLimit.RateLimitMiddleware1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:l

Any thoughts on this one? This appeared to be working fine last week, however is now unusable
Cheers

2

I just pulled the last container and created a super simple API and didn’t see this problem. Can you share what your APIs are doing?

3

They mostly reference other APIs, joining and parsing from different sources and returning a single response.

At the moment I seem to be able to make a single request if I’m lucky, on the console I can see the API server appears to exit - is there an envvar I can enable debug or log I can look at to see why this might be?

info: UniversalAutomation.GroomService[0]
Finished groom job.
info: Universal.Server.Services.ApiProxy[0]
Exited…
fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[1]
An unhandled exception has occurred while executing the request.
Grpc.Core.RpcException: Status(StatusCode=Unavailable, Detail=“Socket closed”)
at Universal.Server.Services.ApiProxy.ExecuteAsync(HttpContext httpContext) in D:\a\universal\universal\src\Universal.Server\Services\ApiProxy.cs:line 357
at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 61
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)

4

Upon reflection I think I may have found the source of the issue - I noticed my pods limits were riding their cpu limits, as someone is now calling some of these APIs quite frequently. After bumping these up, it appears to be stable again.

1 Like