I recently upgraded from version 5.4.3 to version 5.6.7. Apps and scripts are working fine. However, neither of my endpoints are working. I’m getting “The remote server returned an error: (500) Internal Server Error.”
I’m also noticing that sometimes when I make a change to an endpoint i’m getting an error in the UI showing this “API environment is not running! Please ensure your environment is configured correctly” I click away to another section and click back into Endpoints and it does seem to save my changes.
Nobody else is having this issue? I can’t get any endpoint to work. not even a simple hello world. Are there new .net or other dependencies with version 5.6.7 that are different from 5.4.3? I can see the release notes for 5.6.7 but I can’t find where any requirements are listed, if any.
i had similar issue when upgrading from 4.4 do to 5.6.1
try to remove every endpoint from endpoints.ps1 (simply comment them out)
and then try to add one at the time.
one of your api endpoint is making problem probably.
in my case there was a 2 or 3 old endpoints from version v3 i guess (like sayit ,hello and one more)
there were defined in this endpoints.ps1 directly and that was probably causing a problem
I typically do not edit the file itself. I do all my editing from the UI. I’ll backup my files and try deleting the endpoints.
In general there is definitely something wrong with this version. I keep getting this API Environment is not running error. and now i’m also getting frequent errors running scripts. some jobs are failing database is locked errors.
I can see part of the problem now. the changes I make in the UI are not actually being written to the endpoints.ps1 file. this seems to only be happening with endpoints. Everything else seems to be updating and saving just fine.
At the moment, I’m not able to edit endpoints in the UI at all. I keep getting that “API environment is not running! Please ensure your environment is configured correctly” error. So right now I have to manually edit the endpoints.ps1 file and restart the powershell universal service every time which is very annoying.
I started by commenting out everything in each endpoint and slowly adding things back until I got an error. It seems that if I set the environment to powershell 5.1, it causes the endpoints to fail. Unfortunately, if I remove powershll 5.1 it still fails. I think I ran into this issue in the past on prior versions and needed 5.1 in order for my code to work. Is there a reason why 5.1 doesn’t work in version 5.6.7?
Hi, we had similar issues and were able to resolve them by restarting the environments. After that, our endpoints worked again. We are using PS 5.1 and Integrated. I hope this might help you as well.
wow that worked. my endpoints work now. and I am now able to edit the code in the UI and not getting that “API environment is not running” error anymore.
strange that restarting the entire powershell universal service wouldn’t accomplish the same thing. also would be good to know why the environments get into a bad state.
Hi, glad to hear it helped you as well. We also couldn’t determine what exactly caused the API issues, since the server, IIS, etc. were all restarted. It was more of a spontaneous action that ended up making the call work again.
we experienced the phenomenon again today that the Environment Windows PowerShell 5.1 stopped working for no apparent reason. I only noticed it spontaneously once more because no requests came in throughout the day.
We have a similar issue open here. Feel free to contribute more information.
I specifically see issues with PowerCLI causing long delays. That said, we also had an issue with PSModulePath in some environments, but it wouldn’t be intermittent.
Does anyone have API crashes? You would see something in Windows Event Viewer about a .NET app crash.
It sounds like some people have it not starting the first time but then restarting the env fixes it, but also it stops working after some time.
We are looking into it but I don’t know root cause yet. I do not see these issues locally. PSU logs, event logs or any other info people could gather would be helpful.
Additionally, a new issue has arisen: the client credentials are suddenly no longer recognized, even though everything was working previously. The error message is: ‘The server has rejected the client credentials.
[11:53:18 INF] UniversalAutomation.Services.UniversalConfigurationService Reading configuration for Tag
[11:53:18 INF] Microsoft.AspNetCore.Routing.EndpointMiddleware Executing endpoint ‘/_blazor/negotiate’
[11:53:18 INF] Microsoft.AspNetCore.Routing.EndpointMiddleware Executing endpoint ‘/_blazor/negotiate’
[11:53:18 INF] Microsoft.AspNetCore.Routing.EndpointMiddleware Executed endpoint ‘/_blazor/negotiate’
[11:53:18 INF] Microsoft.AspNetCore.Routing.EndpointMiddleware Executed endpoint ‘/_blazor/negotiate’
[11:53:18 INF] Microsoft.AspNetCore.Hosting.Diagnostics Request finished HTTP/1.1 POST https://XXX:8088/_blazor/negotiate?negotiateVersion=1 - 200 316 application/json 383.1649ms
[11:53:18 INF] Microsoft.AspNetCore.Hosting.Diagnostics Request finished HTTP/1.1 POST https://XXX:8088/_blazor/negotiate?negotiateVersion=1 - 200 316 application/json 384.5702ms
[11:53:18 INF] Microsoft.AspNetCore.Hosting.Diagnostics Request finished HTTP/1.1 POST https://XXX:8088/Endpoint - 404 null null 398.2851ms
[11:53:18 INF] Microsoft.AspNetCore.Hosting.Diagnostics Request reached the end of the middleware pipeline without being handled by application code. Request path: POST https://XXX:8088/Endpoint, Response status code: 404
[11:53:18 INF] Microsoft.AspNetCore.Hosting.Diagnostics Request starting HTTP/1.1 GET https://XXX:8088/_blazor?id=z3tuhILP9HtN2NdA7F1qfg - null null
[11:53:18 INF] Microsoft.AspNetCore.Hosting.Diagnostics Request starting HTTP/1.1 GET https://XXX:8088/_blazor?id=GuC7OfIwKae2jiA1RMtcnQ - null null
[11:53:18 INF] Microsoft.AspNetCore.Cors.Infrastructure.CorsService CORS policy execution failed.
[11:53:18 INF] Microsoft.AspNetCore.Cors.Infrastructure.CorsService Request origin https://XXX:8088 does not have permission to access the resource.
[11:53:18 INF] Microsoft.AspNetCore.Cors.Infrastructure.CorsService CORS policy execution failed.
[11:53:18 INF] Microsoft.AspNetCore.Cors.Infrastructure.CorsService Request origin https://XXX:8088 does not have permission to access the resource.
[11:53:18 INF] Microsoft.AspNetCore.Routing.EndpointMiddleware Executing endpoint ‘/_blazor’
[11:53:18 INF] Microsoft.AspNetCore.Routing.EndpointMiddleware Executing endpoint ‘/_blazor’
[11:53:18 INF] Microsoft.AspNetCore.Routing.EndpointMiddleware Executed endpoint ‘/_blazor’
[11:53:18 INF] Microsoft.AspNetCore.Hosting.Diagnostics Request finished HTTP/1.1 GET https://XXX:8088/_blazor?id=z3tuhILP9HtN2NdA7F1qfg - 101 null null 316.252ms
[11:53:18 INF] Microsoft.AspNetCore.Routing.EndpointMiddleware Executed endpoint ‘/_blazor’
[11:53:18 INF] Microsoft.AspNetCore.Hosting.Diagnostics Request finished HTTP/1.1 GET https://XXX:8088/_blazor?id=GuC7OfIwKae2jiA1RMtcnQ - 101 null null 319.885ms
All of my endpoints stopped working again. I’ve tried restarting the puniversal windows service. i’ve also tried restarting each powershell environment. When I try to edit an endpoint I get this error saying that the API environment is not running. When trying to use an endpoint I get this:
Invoke-RestMethod : The underlying connection was closed: An unexpected error occurred on a send.
Application: Universal.Server.exe
CoreCLR Version: 9.0.925.41916
.NET Version: 9.0.9
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AggregateException: One or more hosted services failed to stop. (A task was canceled.) (The operation was canceled.) (The operation was canceled.)
—> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at Universal.Server.Services.Configuration.ConfigurationSystemWatcher.StopAsync(CancellationToken cancellationToken) in D:\a\universal\universal\src\PowerShellUniversal.Configuration\ConfigurationSystemWatcher.cs:line 87
at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List1 exceptions, Func3 operation) --- End of inner exception stack trace --- at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host) at Universal.Server.Program.Main(String[] args) in D:\a\universal\universal\src\Universal.Server\Program.cs:line 37 ---> (Inner Exception #1) System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at System.Threading.CancellationToken.ThrowIfCancellationRequested() at Hangfire.Processing.TaskExtensions.WaitOneAsync(WaitHandle waitHandle, TimeSpan timeout, CancellationToken token) in C:\projects\hangfire-525\src\Hangfire.Core\Processing\TaskExtensions.cs:line 80 at Hangfire.Processing.BackgroundDispatcher.WaitAsync(TimeSpan timeout, CancellationToken cancellationToken) in C:\projects\hangfire-525\src\Hangfire.Core\Processing\BackgroundDispatcher.cs:line 82 at Hangfire.Server.BackgroundProcessingServer.WaitForShutdownAsync(CancellationToken cancellationToken) in C:\projects\hangfire-525\src\Hangfire.Core\Server\BackgroundProcessingServer.cs:line 148 at Hangfire.BackgroundJobServerHostedService.StopAsync(CancellationToken cancellationToken) in C:\projects\hangfire-525\src\Hangfire.NetCore\BackgroundJobServerHostedService.cs:line 134 at UniversalAutomation.StartupService.StopAsync(CancellationToken cancellationToken) in D:\a\universal\universal\src\Universal.Server\Services\StartupService.cs:line 610 at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List1 exceptions, Func3 operation)<—
—> (Inner Exception #2) System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Threading.CancellationToken.ThrowIfCancellationRequested()
at Microsoft.Extensions.Hosting.WindowsServices.WindowsServiceLifetime.StopAsync(CancellationToken cancellationToken)
at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)<—
