Best way to troubleshoot high memory and CPU usage? (and crashes)

Universal Dashboard Universal Dashboard Help
1

I am having a dashboard running that over time use alot of memory and start to use up all the CPU, the dashboard get unresponsive.

It’s started in a PS core 6.1 console.

Would Windows handle this better if it run in an IIS or as a windows service?

2

image
image.png938×537 37.4 KB

So I made a chart on the memory footprint, and as we can see it grow every 1 minut, so it’s the Scheduled endpoints that leak memory.

Do this have anything on it? https://stackoverflow.com/questions/51012481/c-sharp-and-powershell-runspaces-creating-massive-unmanaged-memory-leak

3

The above memory usage are using PS Core, here are running with PS 5.0

image
image.png943×545 37.1 KB

I am using 2 modules beside UD.community:
dbatools
activedirectory

4

I suspect the “leak” is simply standard PowerShell practices wherein we don’t clean up after ourselves, relying on .Net garbage collection and the PowerShell thread going away when we’re done with it to clean up memory.

In UD, Powershell runspaces are being reused, and thus are not going away at the end of an endpoint which would otherwise delete their memory allocation. And normal .Net garbage collection is, in some edge cases, not sufficiently aggressive at cleaning up memory when we are aggressively filling it.

Try nulling out any local variables at the end of memory-intensive endpoints, and finish by “manually” triggering garbage collection.

$Endpoint = New-UDEndpoint -Schedule $Schedule -Endpoint {
    $X = 0..50000000
    $Cache:XCount = $X.Count

    #  Clean up memory
    $X = $Null
    [gc]::Collect()
    }

Thanks,
Tim Curwick

3 Likes
5

This is the exact reason I switched to IIS, performance was way better, but still, having my dashboard running over a few days does the same, but a scheduled task to reset the iis site at midnight fixes that :slight_smile:

6

I made this “auto runner” :slight_smile:

Start-Transcript $psscriptroot\logs\autorun.txt -Append -Force
$InformationPreference = ‘Continue’
while ($true) {
if ($Dashboard.HasExited) {
Write-Information -MessageData “[$(Get-Date -Format ‘dd/MM/yy HH:mm:ss’)] Dashboard Crashed”
$start = $true
}
if ($Start -ne $false) {
Write-Information “[$(Get-Date -Format ‘dd/MM/yy HH:mm:ss’)] Starting Dashboard”
$Dashboard = Start-Process -PassThru -WorkingDirectory $psscriptroot -FilePath pwsh.exe -WindowStyle Minimized -ArgumentList “-file $psscriptroot\invoke-dashboard.ps1”
$start = $false
}
$MemoryUsed = (Get-Process -Id $Dashboard.Id).WorkingSet64 / 1mb
if ($MemoryUsed -gt 1024) {
Write-Information “[$(Get-Date -Format ‘dd/MM/yy HH:mm:ss’)] Dashboard reached memory limit, restarting dashboard”
$Dashboard.Kill()
$Dashboard = $null
$start = $true
}

Need to Fetch before Status to compare against server status

git fetch
$status = git status
if($status -like “Your Branch is behind”) {
Write-Information “[$(Get-Date -Format ‘dd/MM/yy HH:mm:ss’)] Dashboard Updating, restarting dashboard”
$Dashboard.Kill()
$Dashboard = $null
$start = $true
git pull
}
Start-Sleep -Seconds 60
}

also I removed the usage of dbatools module, as that were the one that took most of the memory usage, so now the dashboard can run for 12 hours ish, before it reached 1gb of ram memory.
much better than before.
for the SQL part, I use netstandard 2.0 SMO library: https://github.com/Microsoft/sqltoolsservice

7

This was the main reason i sorta stopped using UD for building a devops api that would run some pester tests…but i see now it was due to how i wasnt cleaning up after myself.

is IIS preferred over running as a service, or maybe in a docker container? I was nearing the point where i was needing to build the routes, but i gave up after i realized my dashboard was bloating after a few days. im gonna start over from scratch.

8

Adam have made alot of improvement to some of the backend systems, so try with newer versions :slight_smile: