Crashing Server 3.6

Product: PowerShell Universal
Version: 3.6.3

Stability of the server has degraded since 3.6. We had to add more memory to the server as was getting git server memory errors and crashing. Today the server lost all the Environments and no scripts would run until the Windows Service was restarted.

Can we easily re-install 3.5.x?
Do we need to do a fresh install?

I was able to remove-psuserver and install-psuserver -version 3.5.5 with no issues so far.

Can you let me know a little about your configuration (git\sql\dashboards etc) ? I’d like to try to reproduce this issue on 3.6.

Name                           Value
----                           -----
PSVersion                      7.3.1
PSEdition                      Core
GitCommitId                    7.3.1
OS                             Microsoft Windows 10.0.20348
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Windows Service installed with Install-PSUServer
Built in DB
Manual Git
git version 2.36.0.windows.1
2 way Sync every minute
One script that calls custom Module functions that runs nightly. Modules installed locally.
Using Environment with PowerShell 7.3.1
FYI: we redirect 5000 to 5463 with a self signed certificate.

Once we went to 3.6 we notice the PowerShell Universal service crashed and never ran the scheduled job.
Log example:

2022-12-20 17:44:02.182 -05:00 [ERR] Failed to sync: An error occurred trying to start process 'git' with working directory 'C:\ProgramData\UniversalAutomation\Repository'. The paging file is too small for this operation to complete.    at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at UniversalAutomation.Git.GitService.RunGitCommand(String args) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\GitService.cs:line 800
   at UniversalAutomation.Git.GitService.Pull() in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\GitService.cs:line 478
   at UniversalAutomation.GitSyncService.TrySync(GitStatus status, Boolean force, GitSettings settings) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\GitSyncService.cs:line 157
   at UniversalAutomation.GitSyncService.Sync(Boolean force) in C:\actions-runner\_work\universal\universal\src\Universal.Server\Services\GitSyncService.cs:line 121

FYI: There is SQL server and IIS on this VM that run other .NET custom apps.

We increased RAM to 48GB. Next issue we started having proxy connection issues to internet sites as the script failed to reach out to our MS SharePoint/Teams site.

We upgraded to 3.6.3 to see if that fixed these issue and that get’s us to the latest issue.
We get the message in the Jobs log

Failed to locate PowerShell version: 7.3.1

And if you check the Environments page it it blank. On the Script you can only select Default for the Environment.

Now we are running 3.5.5 with no other changes and all working again.

I have nothing of value to add other than 3.6.3 was an unholy mess - constant errors, timeouts, etc so I too rolled back. Lesson: Let others be the guinea pigs!

Can either of you share log files from when these issues were happening? Feel free to send to support@ironmansoftware.com

It looks like we are leaking endpoints in dashboards that are likely leading to lots of problems with memory. We’ll be testing a patch and likely releasing sometime tomorrow to address this.

I am still seeing the service crashing on versions 3.5.5 and now on 3.6.4. I will collect some logs and send them to support this week.

1 Like

How is this progressing?

David’s issues seem to be related to other processes running on his machine and not PowerShell Universal itself. He’s either going to move to another server or try to fix the memory issue in the other processes.

3.6.3 did have an issue with endpoint clean up that was resolved in 3.6.4 that could cause excess memory issues and maybe the issues you were seeing with 3.6.3. I would suggest upgrading to 3.6.4.