Adding/Removing variables or save of changes in scripts takes very long time

Hi

I have around 50 scripts so far and similar number of variables (a lot Secret stored in PSUSecretStore). Two way Repo sync is enabled.
Whenever I’m adding or removing variable or save changes in script through PSU - it can take up to few minutes to complete operation. Subjectively situation getting worse.
Server quite heavily used, but is not short in CPU or Memory (both ~50%).

Any ideas what can slow down this activities?

P.S.: I’ve tried to get some info with hangfire (never used it before), but got 500 error. Will appreciate for any info on how to resolve this as well.

We do have an issue with saving variables where it needs to reset the environments for dashboards and APIs so they pick up the new variable changes.

If you have a lot of dashboards or APIs, this can be slow. What could be compounding this is that resetting environments currently causes secrets to be reloaded and if you have a lot, that could cause a slow down because the encryption in the secret store is slow. It needs to decrypt each variable to place it in the environment.

The second issue with secrets has been mitigated in v3 by adding a $Secret scope for secret variables so they are only decrypted when used.

I don’t know why saving scripts would be slow.

To get a definitive answer, we could collect a couple memory dumps of the process while it’s attempting to save stuff. You’ll want to capture several since they will just be points in time. Hopefully, they’ll show a similar location when the slow down is taking place.

Once you have those, you can upload them here and I can look: Dropbox - Submit files

Scrips save time vary a lot. It can be 30 seconds or just a few. Although it’s not very convenient, I’d say that variable add\remove is a bigger issue for me. I’m moving part of my automation from Azure Automation to PSU and I need to add for now 60+ secret variables, then even more. Just tested variable save - it took 2.6-4.2 minutes. For file save - dump creation is longer then save time, so I think we can ignore it for now =)
Dumps are quite big (over 2GB limit of Dropbox) and not compressing really, as I’m storing quite some data in $Cache variables. Instead, I’ve uploaded link to my OneDrive.

Thanks for assistance.

Can you let me know what version you’re running? I’m seeing some behavior I thought had been resolved in a previous version.

My version is 2.10.2

Can you please upgrade to 2.11.1? I don’t think it will necessarily solve your issue but it will certainly reduce the number of runaway threads you have in the process. I think it’s worth trying.

1 Like

I’d say it fixed most of the problem.
On average - secret variables are added about 15s, but can peak at 1.5 minutes.
A bit on a slower side, but I’ve finish adding 60 secret variables and I can leave with 15 second wait.
P.S. Update did fix hangfire as well.

Thanks.

2 Likes