Preserve data between scripts and API's

PowerShell Universal
1

Hi,

I’m trying to migrate from Universal Dashboard to Powershell Universal. In UD - I have set of scheduled and long running endpoints, that fetch data from various systems and store results in $Cache: vasibales. This way - I was able to decouple data collection from calls, as data collection can take anything from 10 to 300 minutes. It works flawlessly on UD for over a year. Now, I’d like to consolidate my efforts in UD and Azure Automation in one tool (without Azure Automation limitations). But I’m not able to find any info on how to preserve and share data between scripts, like $Cache: variables or maybe some new feature.

2

You can now use the server-level cache to store data between scripts. The $Cache scope will work for sharing data if everything is using the integrated environment.

3

Thanks a lot. That covers my question.
Is there any performance penalties between $Cache variables and server-level caching? I’m storing few GB of cached data in such variables.

4

The $Cache variable uses a static dictionary so it’s pretty much storing it directly in memory as live objects. The Set-PSUCache will perform serialization so the objects aren’t live and because of this, you’ll see a performance hit with that method.

For several GBs of data, I would recommend avoiding the server-level cache and use the integrated environment, if possible.

5

Just wanted to report, that Set\Get-PSUCache (and under the hood, I assume, convertto-clixml) is a real bottleneck on even moderate size datasets. 200MB JSON is almost impossible to work on. It consumes over 10GB of RAM and serialization\deserialization takes all available CPU juices. And from time to time I even got OutOfMemory errors in jobs (on 64GB RAM VM).

I know that this is not an issue of PU specifically, but it’d be great to have some general advices on how to efficiently manipulate big data sets, when integrated environment is not an option with $Cache variables. Maybe how to avoid unnecessary serializations\deserializations, make more efficient data transfers between PU core and environments, etc.