Git Synchronization Failed: sporadic errors after updating git settings

PowerShell Universal
1 
Product: PowerShell Universal
Version: 4.4.1

Running a licensed MSI install on Windows Server 2019 Standard.

Hi all,

We recently moved our psu content to a new enterprise github repo and we’re seeing odd sporadic issues around git synchronization failing, as well as our custom modules not being loaded into one of our dashboards/psu apps.

I’ll outline the steps I performed below to swap to the new repo:

  • clone new repo using same ssh keypair for previous repo
  • stop PSU service; set to manual start; restart server
  • rename pre-existing/old repo folder to repository.old
  • rename new repo folder to repository
  • start PSU service

No PSU configs were changed between the old repo and the new repo (besides the git sync config changed through the admin panel).

Everything started up fine and git sync was/is working without issue.

When testing our dashboard/app, we noticed that one of our custom modules fails to be loaded into the dashboard. A restart of the PSU service fixes this, but the error occurs sporadically.
We plan on poking around when this issue occurs next to try to track down why this is happening.

We’re now seeing 2 additional sporadic issues that have popped up in the past few days around git sync.

An error with grpc (we don’t use grpc at all in PSU):

Status(
    StatusCode="Unavailable", 
    Detail="failed to connect to all addresses", 
    DebugException="Grpc.Core.Internal.CoreErrorDetailException: 
    {
        "created":"@1742876235.068000000",
        "description":"Failed to pick subchannel",
        "file":"..\..\..\src\core\ext\filters\client_channel\client_channel.cc",
        "file_line":3218,
        "referenced_errors":[
            {
                "created":"@1742876235.068000000",
                "description":"failed to connect to all addresses",
                "file":"..\..\..\src\core\lib\transport\error_utils.cc",
                "file_line":165,
                "grpc_status":14
            }
        ]
    }"
)

For this issue, I don’t really have a clue how to dig into this. It doesn’t really make sense to me.

And an error with file permissions on .git/config:

Git command failed with exit code 128. 
Errors: 
    error: could not write config file .git/config: 
        Permission denied fatal: could not set 'remote.origin.url' to 'git@companygithub.company.example.com:github-org/psu.git'

For this issue, I’ve compared our old repo .git/config file permissions with our new repo; and they are identical. The service account that PSU is running under has full control of all folders & files under the new repo, the exact same as the old repo.

All of the issues occur sporadically.

If anyone has experienced any of the above issues and has any thoughts/recommendations in terms of tracking down the core issue, please chime in!

Thanks,
pb

1 Like
2

Is it possible that you have some kind of scheduled task or another service installed from the old repo connectivity that’s trying to access the .git/config file at the same time that PSU is attempting to access it? That would explain the sporadic behavior.

As for the gRPC error(s), you say you aren’t using gRPC at all. Does that mean that you’re using the parameter in the appsettings.json file to put PSU into Permissive or Integrated mode, to remove the security requirements, or did you set "EnableWebSockets": false to entirely disable Web Sockets?

3

Thanks for replying!

We have no scheduled tasks interacting with the .git/config file.

Sorry I could have been more specific when saying

we don’t use grpc at all in PSU

What I meant is that we don’t interact with PSU via gRPC at all. We haven’t modified any settings in appsettings.json to disable PSU internal gRPC usage.

In re-reading my post, I know I outlined quite a few issues.

The main issue is the sporadic failing to load our custom module into one of our core dashboards/apps. The module in question simply wraps calls to various UD/UniversalDashboard cmdlets to build UI components in our dashboards/apps.
This issue has only started to occur since changing repos & git sync config in PSU. And just to reiterate, the content between the old repo & new repo is the exact same (the reason for swapping is due to some branch rulesets that we’re unable to modify due to silly internal company processes).

4

That’s not quite what I was asking. If you have another Git-related process (i.e., another Git client) that’s trying to do a sync or something on the same repository path (since I assume you’re using the same repo paths as you were with the old Git service), that could cause a file access conflict.

Maybe we’re saying the same thing with different phrasing. I don’t know.

5

We don’t have any other processes using git outside of PSU on the server.

6

Are you using any Git commands within any of your scripts inside of PSU, in addition to PSU’s actual Git settings? For example, in some of my scripts I have git add commands to add files downloaded through some of the jobs, because PSU would otherwise not know about them and they wouldn’t be part of the repository as they were not created by PSU itself. Initially I had some other Git commands but they were causing issues (race conditions) with the index files being locked by other jobs, so I had to remove the other commands and leave only git add.

7

No we don’t have anything of that sort. Nothing else uses git.exe on the server; only the PSU git sync.

8

Okay. Sorry for asking so many times about Git - I just want to be thorough.

Are your Git settings defined in appsettings.json or are they set inside of PSU directly? If directly, it’s possible there could be a conflicting entry in the database from the old repository that’s causing issues (conflicts) with the new repository settings. If you have access to the PSU server host, you can get a copy of DB Browser for SQLite (assuming your installation is using SQLite and not something else) and view the GitSettings table to see whether or not there are old/bad entries in there - in which case, you could remove the entry from the table.

9

The git settings are defined through the admin portal UI, not in appsettings.json. I took a look at the dbo.GitSettings table (we’re using MSSQL) and the settings are aligned with what we’ve set in the UI.

I’m not sure that git sync is our problem here; I’ve only included the context around what was changed within PSU now that we’ve started to see the issue with our module sometimes not being loaded into our dashboard/app.

10

That’s fair. I just wanted to see if it was maybe related to the other issue.

For the gRPC issue, you could try adjusting the settings in the appsettings.json file that I mentioned previously, to either change the security requirements to Integrated or Permissive. You can read more about the settings at Module | PowerShell Universal