Universal Dashboard becomes very slow after consuming about 4 GB of memory

1

Hi everyone,

I have decided to share my test of UD performance hopefully we can address this issue soon.

i have UD hosted in IIS server with 16GB of memory with xeon 4 core processor so there is a plenty of power there.

once i start UD its very fast very responsive no issues.

i have 6 cache endpoint scheduled to collect services , Tasks, etc…

when UD runs the memory starts to build up based on the data generated by these endpoints scheduled.
the time i chose in these endpoints scheduled is the factor that controls how fast the server memory consumption will be.
so for example running these scheduled every 30 seconds is different than running them every 1 hour.

the issue here is we need to run these scheduled every x seconds to be able to get the updated status of an on click command.

i have decided to try to test running these scheduled every 1 day to monitor server performance and the results was as expected UD keep running at full performance since memory consumption was 345MB only but the down side i wont get updated status obviously after an on click command.

BoSen29 created an Invoke-UDEvent which is a great command to use but the problem is after on click when getting the updated status in the grid the button onclick wont do anything.

The main problem i have noticed is if i kept the scheduled endpoints running every x seconds , once the server memory consumption reach 34% (8GB of 16GB) UD is very slow and start seeing these errors in the log file.

02:07:59 [Error] Quartz.Core.JobRunShell Job DEFAULT.1e650830-0591-4994-9d17-23f6ff9d3bd7 threw an unhandled Exception:
02:08:02 [Error] Quartz.Core.ErrorLogger Job DEFAULT.1e650830-0591-4994-9d17-23f6ff9d3bd7 threw an exception.

can we get invoke-udevent command fixed so it will behave the same way endpoint scheduled calling the cache endpoint this way we can keep UD running at best performance and we dont have to setup a scheduled to run every x seconds instead will keep it for example every 1 day and once any user click a button invoke-udevent will call that scheduled endpoint and update the status in the grid.

Thank you

2

Update:

I think the issue related to the following

when we defined an endpoint in UD we are doing it this way

$Services = New-UDEndpoint -Id Test -Schedule $Schedule1 -Endpoint {…}
invoke-udevent -id Test -schedule will call the endpoint itself which means the $Services value didnt get updated.

3

error getting in UD

image
image772×67 3.23 KB

4

Update

i have tried to give the button an ID that match object name and based on testing the error dont show again but clicking the button first time will do the work for example
first click will stop a service > grid updated with service stopped
second click to start the stopped service again > stop service executed again

5

Hi
I think we’re going to need more detail. I’ve never seen this issue, don’t have a problem with memory and yet I have many endpoints, scheduled endpoints, cached data, doing various things from pulling sql data to running actual scripts checking servers, services etc.

Really, before we can just jump to conclusions I think we’d need to see your actual code or at least an example of.
It depends on what you are doing and how you are doing it to get to the root cause of your problem.
It would be also good to see if we can replicate the same issue on another environment :slight_smile:

6

@insomniacc

here is a sample code

Get-UDDashboard | Stop-UDDashboard

$Schedule1 = New-UDEndpointSchedule -Every 1 -Day

$Services = New-UDEndpoint -Schedule $Schedule1 -Id "MyTest" -Endpoint {

$Cache:Services = (Get-Service -ComputerName "server1" -Name "*Certification*") | Select-Object Name, Status | Sort-Object Name | ForEach-Object { 
                
                
    $FontColor = 'Green'

    if ($_.Status -ne 'Running') {
                    
        $FontColor = 'Red'
    }
                
                
                
    [PSCustomObject]@{
        Name   = $_.Name
        Status = New-UDElement -Tag 'div' -Attributes @{ style = @{ color = $FontColor } } -Content { $_.Status.ToString() }
        Select = if ($_.Status -eq 'Stopped') {

            New-UDTooltip -Type info -Effect float -Place top -TooltipContent { "Start service!" } -Content {
                New-UDButton -BackgroundColor "Red" -Text "Start" -OnClick {
                    Get-Service -ComputerName "server1" -Name $_.Name | Start-Service
                    Invoke-UDEvent -Id MyTest -scheduled
                    Show-UDToast -Message "Service Started!" -MessageColor Green -Title $_.Name -Position topCenter -Duration 2500
                }
            }

        }
        else {

            if ($_.Status -eq 'Running') {

                New-UDTooltip -Type info -Effect float -Place top -TooltipContent { "Stop service!" } -Content {
                    New-UDButton -BackgroundColor "#26a69a" -Text "Stop" -Icon stop_circle -OnClick {
                        Get-Service -ComputerName "server1" -Name $_.Name | Stop-Service
                        Invoke-UDEvent -Id MyTest -scheduled
                        Show-UDToast -Message "Service Stopped!" -MessageColor Red -Title $_.Name -Position topCenter -Duration 2500
                    
                    }
                }
            }
        }
    }
}
}

$HomePage = New-UDPage -Name "Home" -Icon home -Content {
New-UDCard -Title "Welcome"

```
New-UdRow {
    New-UdColumn -Size 6 -Content {
        New-UdRow {
            New-UdColumn -Size 12 -Content {
                New-UdTable -Title "Server Information" -Headers @(" ", " ") -Endpoint {
                    @{
                        'Computer Name'         = $env:COMPUTERNAME
                        'Operating System'      = (Get-CimInstance -ClassName Win32_OperatingSystem).Caption
                        'Total Disk Space (C:)' = (Get-CimInstance -ClassName Win32_LogicalDisk -Filter "DeviceID='C:'").Size / 1GB | ForEach-Object { "$([Math]::Round($_, 2)) GBs " }
                        'Free Disk Space (C:)'  = (Get-CimInstance -ClassName Win32_LogicalDisk -Filter "DeviceID='C:'").FreeSpace / 1GB | ForEach-Object { "$([Math]::Round($_, 2)) GBs " }
                    }.GetEnumerator() | Out-UDTableData -Property @("Name", "Value")
                }
            }
        }
        New-UdRow {
            New-UdColumn -Size 3 -Content {
                New-UdChart -Title "Memory by Process" -Type Doughnut -RefreshInterval 5 -Endpoint {
                    Get-Process | ForEach-Object { [PSCustomObject]@{ Name = $_.Name; WorkingSet = [Math]::Round($_.WorkingSet / 1MB, 2) } } | Out-UDChartData -DataProperty "WorkingSet" -LabelProperty Name
                } -Options @{
                    legend = @{
                        display = $false
                    }
                }
            }
            New-UdColumn -Size 3 -Content {
                New-UdChart -Title "CPU by Process" -Type Doughnut -RefreshInterval 5 -Endpoint {
                    Get-Process | ForEach-Object { [PSCustomObject]@{ Name = $_.Name; CPU = $_.CPU } } | Out-UDChartData -DataProperty "CPU" -LabelProperty Name
                } -Options @{
                    legend = @{
                        display = $false
                    }
                }
            }
            New-UdColumn -Size 3 -Content {
                New-UdChart -Title "Handle by Process" -Type Doughnut -RefreshInterval 5 -Endpoint {
                    Get-Process | Out-UDChartData -DataProperty "HandleCount" -LabelProperty Name
                } -Options @{
                    legend = @{
                        display = $false
                    }
                }
            }
            New-UdColumn -Size 3 -Content {
                New-UdChart -Title "Threads by Process" -Type Doughnut -RefreshInterval 5 -Endpoint {
                    Get-Process | ForEach-Object { [PSCustomObject]@{ Name = $_.Name; Threads = $_.Threads.Count } } | Out-UDChartData -DataProperty "Threads" -LabelProperty Name
                } -Options @{
                    legend = @{
                        display = $false
                    }
                }
            }
        }
        New-UdRow {
            New-UdColumn -Size 12 -Content {
                New-UdChart -Title "Disk Space by Drive" -Type Bar -AutoRefresh -Endpoint {
                    Get-CimInstance -ClassName Win32_LogicalDisk | ForEach-Object {
                        [PSCustomObject]@{ DeviceId = $_.DeviceID;
                            Size                    = [Math]::Round($_.Size / 1GB, 2);
                            FreeSpace               = [Math]::Round($_.FreeSpace / 1GB, 2); 
                        } } | Out-UDChartData -LabelProperty "DeviceID" -Dataset @(
                        New-UdChartDataset -DataProperty "Size" -Label "Size" -BackgroundColor "#80962F23" -HoverBackgroundColor "#80962F23"
                        New-UdChartDataset -DataProperty "FreeSpace" -Label "Free Space" -BackgroundColor "#8014558C" -HoverBackgroundColor "#8014558C"
                    )
            } -Options @{
                scales = @{
                    yAxes = @(@{
                            min = 0
                        })
                }
            }
        }
    }
}
New-UdColumn -Size 6 -Content {
    New-UdRow {
        New-UdColumn -Size 6 -Content {
            New-UdMonitor -Title "CPU (% processor time)" -Type Line -DataPointHistory 20 -RefreshInterval 5 -ChartBackgroundColor '#80FF6B63' -ChartBorderColor '#FFFF6B63'  -Endpoint {
                try {
                    Get-Counter '\Processor(_Total)\% Processor Time' -ErrorAction SilentlyContinue | Select-Object -ExpandProperty CounterSamples | Select-Object -ExpandProperty CookedValue | Out-UDMonitorData
                }
                catch {
                    0 | Out-UDMonitorData
                }
            }
        }
        New-UdColumn -Size 6 -Content {
            New-UdMonitor -Title "SQL CPU (% processor time)" -Type Line -DataPointHistory 20 -RefreshInterval 5 -ChartBackgroundColor '#80FF6B63' -ChartBorderColor '#FFFF6B63'  -Endpoint {
                try {
                    Get-Counter '\Processor(_Total)\% Processor Time' -ComputerName sql -ErrorAction SilentlyContinue | Select-Object -ExpandProperty CounterSamples | Select-Object -ExpandProperty CookedValue | Out-UDMonitorData
                }
                catch {
                    0 | Out-UDMonitorData
                }
            }
        }
        New-UdColumn -Size 6 -Content {
            New-UdMonitor -Title "Memory (% in use)" -Type Line -DataPointHistory 20 -RefreshInterval 5 -ChartBackgroundColor '#8028E842' -ChartBorderColor '#FF28E842'  -Endpoint {
                try {
                    Get-Counter '\memory\% committed bytes in use' -ErrorAction SilentlyContinue | Select-Object -ExpandProperty CounterSamples | Select-Object -ExpandProperty CookedValue | Out-UDMonitorData
                }
                catch {
                    0 | Out-UDMonitorData
                }
            }
        }
        New-UdColumn -Size 6 -Content {
            New-UdMonitor -Title "SQL Memory (% in use)" -Type Line -DataPointHistory 20 -RefreshInterval 5 -ChartBackgroundColor '#8028E842' -ChartBorderColor '#FF28E842'  -Endpoint {
                try {
                    Get-Counter '\memory\% committed bytes in use' -ComputerName sql -ErrorAction SilentlyContinue | Select-Object -ExpandProperty CounterSamples | Select-Object -ExpandProperty CookedValue | Out-UDMonitorData
                }
                catch {
                    0 | Out-UDMonitorData
                }
            }
        }
    }
    New-UdRow {
        New-UdColumn -Size 6 -Content {
            New-UdMonitor -Title "Network (IO Read Bytes/sec)" -Type Line -DataPointHistory 20 -RefreshInterval 5 -ChartBackgroundColor '#80E8611D' -ChartBorderColor '#FFE8611D'  -Endpoint {
                try {
                    Get-Counter '\Process(_Total)\IO Read Bytes/sec' -ErrorAction SilentlyContinue | Select-Object -ExpandProperty CounterSamples | Select-Object -ExpandProperty CookedValue | Out-UDMonitorData
                }
                catch {
                    0 | Out-UDMonitorData
                }
            }
        }
        New-UdColumn -Size 6 -Content {
            New-UdMonitor -Title "SQL Network (IO Read Bytes/sec)" -Type Line -DataPointHistory 20 -RefreshInterval 5 -ChartBackgroundColor '#80E8611D' -ChartBorderColor '#FFE8611D'  -Endpoint {
                try {
                    Get-Counter '\Process(_Total)\IO Read Bytes/sec' -ComputerName sql -ErrorAction SilentlyContinue | Select-Object -ExpandProperty CounterSamples | Select-Object -ExpandProperty CookedValue | Out-UDMonitorData
                }
                catch {
                    0 | Out-UDMonitorData
                }
            }
        }
        New-UdColumn -Size 6 -Content {
            New-UdMonitor -Title "Disk (% disk time)" -Type Line -DataPointHistory 20 -RefreshInterval 5 -ChartBackgroundColor '#80E8611D' -ChartBorderColor '#FFE8611D'  -Endpoint {
                try {
                    Get-Counter '\physicaldisk(_total)\% disk time' -ErrorAction SilentlyContinue | Select-Object -ExpandProperty CounterSamples | Select-Object -ExpandProperty CookedValue | Out-UDMonitorData
                }
                catch {
                    0 | Out-UDMonitorData
                }
            }
        }
        New-UdColumn -Size 6 -Content {
            New-UdMonitor -Title "SQL Disk (% disk time)" -Type Line -DataPointHistory 20 -RefreshInterval 5 -ChartBackgroundColor '#80E8611D' -ChartBorderColor '#FFE8611D'  -Endpoint {
                try {
                    Get-Counter '\physicaldisk(_total)\% disk time' -ComputerName sql -ErrorAction SilentlyContinue | Select-Object -ExpandProperty CounterSamples | Select-Object -ExpandProperty CookedValue | Out-UDMonitorData
                }
                catch {
                    0 | Out-UDMonitorData
                }
            }
        }
    }
    New-UdRow {
        New-UdColumn -Size 12 {
            New-UdGrid -Title "Processes" -Headers @("Name", "ID", "Working Set", "CPU") -Properties @("Name", "Id", "WorkingSet", "CPU") -AutoRefresh -RefreshInterval 60 -Endpoint {
                Get-Process | Select-Object Name, ID, WorkingSet, CPU | Out-UDGridData
             }
          }
       }
    }
 }
```

}

$Page1 = New-UDPage -Name "Services" -Icon server -Content {

```
New-UDElement -Id "Services" -Tag div4 -EndPoint {

    New-UDGrid -Id "Grid" -Title "Services" -Headers @("Name", "Status", "Select") -Properties @("Name", "Status", "Select") -Endpoint {
    
        $Cache:Services | Out-UDGridData
           
    }
                             
}  
                    
New-UDButton -BackgroundColor "#26a69a" -Text "Refresh" -OnClick {
    Sync-UDElement -Id "Grid" -Broadcast       
} 
```

}

Start-UDDashboard -Endpoint @($Services) -Content {
New-UDDashboard -Pages @($HomePage, $Page1) -Title "Service Manager" -Color '#FF050F7F'
} -Port 1000 -Name ServiceM -AutoReload

also the idea here is simple, whenever you use cache in UD in a repeated schedule your memory consumption start to increase so the goal here is to eliminate this issue where you dont have to keep generating cache over and over if you can call the schedule manually and maintain your endpoints.

in my environment am using UD community in IIS on a win2019 server, because of the memory consumption occurred by cache objects loop i had to schedule an app pool recycle at 10GB but didnt notice that after hitting 34% of RAM consumption UD will become very slow.

7

I am the same experience, but not looked that much into it.

My dashboard are also using more and more memory.
Even then it should only be using like 300-400mb og memory. ( it start with that much, and over time get more and more. )

It will get to a stall in performance, and i have to recycle the app pool.

8

I restart my webserver nightly to cope with the memory growth. I find this works well for me since i can just schedule it.

9

So in my case am doing the same but my issue is the button endpoint change after refresh using invoke-udevent and the buttons becomes unusable.
if you are using cache and you dont have elements such as buttons this command will help you improve the performance of your dashboard.

10

just to summarize the goal here

having an endpoint cache that runs every 10 seconds and collect data will consume memory to a point where when it reaches 35 % of server available memory UD performance get affected.

need a solution to be able to execute the endpoint manually and retain elements like the button functionality.

11

@wsl2001
I’ll give the Invoke-UDEvent another look today, and verify the bug.

The memory issues you guys are describing however, i haven’t seen since 2.4 myself.
Resolution to my memory issues in my early UD-using days were to ALWAYS do a “| select-object property1, property2 | out-udgriddata” in all my grids and etc.
I found that sending non specified properties to the grid, would result in a memory leak. This might have been patched, but i’ve always done this since the 2.3 days.

12

@BoSen29
good to see you man , as you have mentioned am using the same idea you just said but here is the issue

am using an endpoint cache to collect services on the system and i cannot use a schedule to run this endpoint once a day , i need it to run every 10 seconds at least to update the status when a user action a start/stop service.

now a s test when i change the schedule to run 1 a day no memory consumption at all but when user action start/stop service nothing will be updated on the grid since you are still reading from the first cache you run once.
when executing the cache every 10 seconds the memory consumption will start build up because a new data is generated every 10 seconds

13

Calling invoke-udevent apparently is not registering the button back in UD even if i give the button an ID

image
image772×67 3.43 KB