500 Error After Git Sync

LiquoriChris · August 30, 2021, 7:01pm

Product: PowerShell Universal
Version: 2.2.1
Install: MSI

Hello,

When committing changes to a repository, I am seeing a 500 error after attempting to sync with PSU. This happens on both One and Two way sync. Below is the error I am seeing

2021-08-30T14:33:36.2073263-04:00 0HMBBSFVF3DMP:000000C5 [ERR] An exception was thrown attempting to execute the error handler. (e55ade05)
System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Universal.Server.Startup.<>c.<Configure>b__7_1(IDatabase database, HttpContext context) in D:\a\universal\universal\src\Universal.Server\Startup.cs:line 378
   at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 71
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
2021-08-30T14:33:36.2090286-04:00 0HMBBSFVF3DMP:000000C5 [ERR] Connection id ""0HMBBSFVF3DMP"", Request id ""0HMBBSFVF3DMP:000000C5"": An unhandled exception was thrown by the application. (560e7d32)
System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Universal.Server.Startup.<>c.<Configure>b__7_1(IDatabase database, HttpContext context) in D:\a\universal\universal\src\Universal.Server\Startup.cs:line 378
   at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 71
   at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
   at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
   at AspNetCoreRateLimit.RateLimitMiddleware`1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
2021-08-30T14:33:36.2091994-04:00 0HMBBSFVF3DMP:000000C5 [INF] Request finished HTTP/2 GET https://<redacted>/admin/automation/scripts - - - 500 0 - 5.8444ms (791a596a)
2021-08-30T14:33:40.7626794-04:00  [INF] Application is shutting down... (90f52c38)

Using MSI Install, after restarting the Windows Service, the application runs normally with the changes made.

Does anyone have a suggestion?

Thanks

adam · August 30, 2021, 7:30pm

I was able to reproduce this with a blank authentication.ps1. Do you have an authentication.ps1 file in your git repo? If not, can you try adding one? If you are using some other kind of auth, you can just return an error message.

Set-PSUAuthenticationMethod -ScriptBlock {
    New-PSUAuthenticationResult -ErrorMessage 'Bad username or password'
}

I also saw this error when I had a syntax error in my authentication.ps1.

I’ll open an issue for this.

LiquoriChris · August 30, 2021, 7:35pm

Hey @adam

I do not have authentication.ps1 in the repo as we are using a different kind of auth. I will add the authentication.ps1 file to the repo and test it out.

Thanks

LiquoriChris · August 30, 2021, 8:12pm

@adam

Adding the authentication did the trick.

Thanks

dkkazak · September 5, 2021, 3:17am

I am also having this issue that whenever a change I make gets synced to the server via GIT that my dashboard crashes and won’t start with a 500 error. If I manually restart the Windows service, everything comes back up. I originally didn’t have an authentication.ps1 file, so did add one, but it is still happening. Any ideas on this? I am running 2.2.1.

adam · September 6, 2021, 1:43am

This is a bug. It’s been resolved in 2.3 nightly and will be part of that build which is releasing on the 14th. You can also download the latest nightly build to get up and running in your environment if you would like.

dkkazak · September 6, 2021, 2:21am

I did download the 2.3 nightly build and attempted another update, but still getting the error. Here is what I have in the logs:
2021-09-05T20:18:41.5325843-06:00 0HMBGRH7ME6G4:0000000E [ERR] An unhandled exception has occurred while executing the request. (48a46595)
System.InvalidOperationException: Sequence contains no elements
at System.Linq.ThrowHelper.ThrowNoElementsException()
at System.Linq.Enumerable.First[TSource](IEnumerable1 source) at UniversalAutomation.SettingsController.Get() in D:\a\universal\universal\src\UniversalAutomation\Controllers\SettingsController.cs:line 66 at lambda_method687(Closure , Object , Object[] ) at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync() at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync() --- End of stack trace from previous location --- at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker) at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger) at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult) at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context) at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 113 at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext) at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider) at Universal.Server.Middleware.SwaggerAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\universal\universal\src\Universal.Server\Middleware\SwaggerAuthMiddleware.cs:line 26 at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<<UseMiddlewareInterface>b__1>d.MoveNext() --- End of stack trace from previous location --- at AspNetCoreRateLimit.RateLimitMiddleware1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi)
at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task)
at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication1 application) 2021-09-05T20:18:41.5337978-06:00 0HMBGRH7ME6G4:0000000E [WRN] No exception handler was found, rethrowing original exception. (61fc7fab) 2021-09-05T20:18:41.5356524-06:00 0HMBGRH7ME6G4:0000000E [ERR] Connection id ""0HMBGRH7ME6G4"", Request id ""0HMBGRH7ME6G4:0000000E"": An unhandled exception was thrown by the application. (560e7d32) System.InvalidOperationException: Sequence contains no elements at System.Linq.ThrowHelper.ThrowNoElementsException() at System.Linq.Enumerable.First[TSource](IEnumerable1 source)
at UniversalAutomation.SettingsController.Get() in D:\a\universal\universal\src\UniversalAutomation\Controllers\SettingsController.cs:line 66
at lambda_method687(Closure , Object , Object[] )
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncObjectResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()
— End of stack trace from previous location —
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Universal.Server.Middleware.RoutingMiddleware.Invoke(HttpContext httpContext, IPolicyEvaluator policyEvaluator) in D:\a\universal\universal\src\Universal.Server\Middleware\RoutingMiddleware.cs:line 113
at Swashbuckle.AspNetCore.SwaggerUI.SwaggerUIMiddleware.Invoke(HttpContext httpContext)
at Swashbuckle.AspNetCore.Swagger.SwaggerMiddleware.Invoke(HttpContext httpContext, ISwaggerProvider swaggerProvider)
at Universal.Server.Middleware.SwaggerAuthenticationMiddleware.InvokeAsync(HttpContext context, RequestDelegate next) in D:\a\universal\universal\src\Universal.Server\Middleware\SwaggerAuthMiddleware.cs:line 26
at Microsoft.AspNetCore.Builder.UseMiddlewareExtensions.<>c__DisplayClass6_1.<b__1>d.MoveNext()
— End of stack trace from previous location —
at AspNetCoreRateLimit.RateLimitMiddleware1.Invoke(HttpContext context) in D:\a\universal\universal\src\AspNetCoreRateLimit\Middleware\RateLimitMiddleware.cs:line 109 at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.HandleException(HttpContext context, ExceptionDispatchInfo edi) at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>g__Awaited|6_0(ExceptionHandlerMiddleware middleware, HttpContext context, Task task) at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.ResponseCompression.ResponseCompressionMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context) at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication1 application)
2021-09-05T20:18:41.5357160-06:00 0HMBGRH7ME6G4:0000000E [INF] Request finished HTTP/1.1 GET http://lapwru:5000/api/v1/settings - - - 500 0 - 6.9610ms (791a596a)

Let me know if you need anything else to troubleshoot this.

adam · September 6, 2021, 2:00pm

Let me try to reproduce this myself some more. Prior to the fix, I was seeing this exact error.

Can you let me know the current state of your git repo?

Do you have a remote that you’ve successfully sync’d before and now it shows this error after a pull? Or do you have an empty folder you are trying to sync?

Never mind: I was able to reproduce this on the latest nightly.

dkkazak · September 6, 2021, 10:59pm

So you know, this was my first foray into using GIT replication. It took a few fits and starts for me to get it going. I eventually uninstalled everything (Windows MSI), deleted the ProgramData folders, reinstalled 2.2.1 and then added things back into it. Sync then seems to happen, but does crash everything when it tries to restart the dashboard (on autodeploy). Let me know if you need anything else to help troubleshoot this.

Thanks!

adam · September 6, 2021, 11:51pm

Thanks for the info. I’m seeing it happen after changing anything and after a git push occurs so what you’re seeing makes sense. I think I’m narrowing in on a fix for that. I’ll keep you updated but I hope to have something tomorrow that will be viable.

adam · September 7, 2021, 3:21am

I think I have a fix for this and have been able to successfully sync with git in numerous scenarios. Feel free to give the next nightly a shot to see if you have a better experience.

dkkazak · September 7, 2021, 10:33pm

Yes, that looks like it is working. Thanks for the quick fix. I’ll keep testing it, but looks good so far.

dkkazak · September 8, 2021, 11:24pm

For the most part, it is working. But, sometimes when it syncs up, the UniversalServer is crashing. Here is what I see in the event logs:
Application: Universal.Server.exe
CoreCLR Version: 5.0.721.25508
.NET Version: 5.0.7
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter ‘chunkLength’)
at System.Text.StringBuilder.ToString()
at Universal.Server.Services.DashboardProxyIntegrated.LogFlushThread() in D:\a\universal\universal\src\Universal.Server\Services\DashboardProxyIntegrated.cs:line 463
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
— End of stack trace from previous location —
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

I can pull up additional logs if that would be helpful. I have had this happen two or three times today (and I am doing some active development, so lots of changes I am testing).

I am using the nightly build from 9/7.

adam · September 8, 2021, 11:42pm

Thanks for the feedback. I’ve opened an issue for this and will look into it tomorrow.

Topic		Replies	Views
Service is running, but HTTP 500 error PowerShell Universal	1	123	June 20, 2024
HTTP Error 502.5 - ANCM Out-Of-Process Startup Failure PowerShell Universal	10	1287	June 29, 2021
Bug: Creating new sched: Sequence contains no elements PowerShell Universal	2	273	June 9, 2023
Git sync error (was: PSU will no longer start in Azure) PowerShell Universal	3	406	July 25, 2022
Dashboards not loading after upgrading to 2.9.0 from 2.8.0 PowerShell Universal	21	879	March 11, 2022

500 Error After Git Sync

Related topics