OutOfMemoryException

PowerShell Universal
1

I have a dashboard that is running some commands from a 3rd-party PowerShell Module (Unity-PowerShell). I can run many of these commands with no problem, but one errors out. In the dashboard log I see:

[06-08-21 01:41:25 PM] An error occurred: Exception of type 'System.OutOfMemoryException' was thrown.
Endpoint: bdd5520c-dd2d-4ae3-a206-b27a94a72894
Session: b231c37b-4e63-40b3-a507-a442d4d80909
File: 
Endpoint Start Line: 182
Endpoint End Line: 227
Stack Trace: at <ScriptBlock>, <No file>: line 34
 
[06-08-21 01:41:25 PM] An error occurred: The remote server returned an error: (422) Unprocessable Entity.
Endpoint: bdd5520c-dd2d-4ae3-a206-b27a94a72894
Session: b231c37b-4e63-40b3-a507-a442d4d80909
File: 
Endpoint Start Line: 182
Endpoint End Line: 227
Stack Trace: at Send-UnityRequest, <No file>: line 33
at Set-UnityLUN<Process>, <No file>: line 212
at <ScriptBlock>, <No file>: line 35

Any idea what might be happening? The code works when I run it in PowerShell directly on the PSU server.

Product: PowerShell Universal
Version: 1.5.18
2

Is it the Send-UnityRequest command causing this or another one and does the dashboard process have super high memory usage when you are using it?

3

Dashboard memory use starts at 153MB and pretty much stays there even after the error.

The last command my dashboard runs before the error is Set-UnityLUN. It looks like Send-UnityRequest is a helper function it calls.

4

Can you try to capture the .NET stack trace for this?

try {
# the command here
} catch {
$_.Exception.StackTrace | Out-File .\temp.txt
}

I’m just wondering if that will narrow down exactly what might be causing it.

5

Hey, sorry it’s been awhile. I thought it might be a quirk of how I had set up my dashboard, since I was unable to reproduce if I stripped the page down to a single button that performed an action. So I refactored to make the page simpler - now this time it’s another command that is giving the same error.

I inserted the code you mention above and here is what I got in the stack trace:

   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame)
6

I just searched for OutOfMemoryException in this forum to see if anyone else had ever received the exception, and found this thread.

It happened to me overnight. A job that was supposed to run this morning (about half an hour ago) threw it, and I couldn’t get back into PSU without restarting the Azure App Service.

I’ve checked the metrics on the app service, and the memory working set didn’t get over 2.5 GB in the past 24 hours. Very strange.

7

I feel like something’s up with memory usage. This is our working set over the past four hours, showing max usage each minute:

image
image1028×326 12.9 KB

No scripts have run. The only activity PSU has done at all is automatically refreshing the “jobs” page when I switch back to the tab. You can see a definite trend upwards in memory usage.

This is what it looks like over the past 30 days:

image
image1287×627 56.3 KB

The big drops are when the service crashes and I have to restart it.

Any clues, @adam? A leak somewhere? I know these things are bloody hard to find.

Update: Our PSU instance is now up to 1.2 GB. Only a few jobs have been run - there is just organic growth between jobs. I’m almost certain it’s something scheduled in the Hangfire back end. Will upgrade to 3.4.3 next week and enable manual Git sync to see if that’s the culprit.

8

OK, I’m running 3.4.3 now with manual Git sync mode enabled. No scripts are scheduled to run in the next four hours, so I will monitor the memory usage and see if it stays flat. Fingers crossed!