Problems accessing the keyvault at weekends

Product: PowerShell Universal
Version: 4.3.4

Hello everyone,

I need your help because I don’t know where to look and especially where this can come from.
Powershell Universal is running under Kubernetes version 4.3.4 with the image “ironmansoftware/universal:4.3.4-ubuntu-20.04”.
For Azure identity, we use Federated credentials in a Managed Identity.
The secrets are stored in an Azure keyvault (credentials and classic string).

Now for my problem. Every weekend, without exception, access to the secrets in the keyvault doesn’t work.
And then, as if by magic, on Monday morning, everything is OK. What’s more, if we reboot the pod at the weekend, everything works again.

This is very problematic as we have jobs that trigger on-call calls (jobs that need credentials).

I’ve tried setting up a job schedule for reconnection to the keyvault, but no change.

My scheduled job

Connect-AzAccount -Subscription $YvarSubscriptionIdTS -TenantId $YvarTenantId -ServicePrincipal -ApplicationId $env:AZURE_CLIENT_ID -FederatedToken $(Get-Content  $env:AZURE_FEDERATED_TOKEN_FILE -raw) -InformationAction Ignore -WarningAction SilentlyContinue

Write-Output "Init Azure Connection for Keyvault `"$KVaultName`" for Secret Variables"

Register-SecretVault -ModuleName Az.KeyVault -Name AzureKeyVault -VaultParameters @{ 
    AZKVaultName = $KVaultName
    SubscriptionId = $YvarSubscriptionIdTS 
} -AllowClobber

And here is the initialize.ps1 script

Import-Module "Az.Accounts"
Import-Module "Az.KeyVault"

$subscription = "xxxxxxxxxxxxxxxxxxxxxxxx"
Connect-AzAccount -Subscription "xxxxxxxxxxxxxxxxxxx" -TenantId "xxxxxxxxxxxxxxxxxxx" -ServicePrincipal -ApplicationId $env:AZURE_CLIENT_ID -FederatedToken $(Get-Content  $env:AZURE_FEDERATED_TOKEN_FILE -raw)
#Disable-AzContextAutosave -Scope Process
#Set-AzContext -Subscription $subscription

Register-SecretVault -ModuleName Az.KeyVault -Name AzureKeyVault -VaultParameters @{ 
    AZKVaultName = 'AKV-TS-YOUTILITY-WEU'
    SubscriptionId = $subscription
} -AllowClobber

Here’s the error message (nothing to write home about)

[09:00:15 ERR] Failed to read secret: Your Azure credentials have not been set up or have expired, please run Connect-AzAccount to set up your Azure credentials.
A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-06-29T07:00:15.1205639Z, assertion valid from 2024-06-25T02:05:52.0000000Z, expiry time of assertion 2024-06-25T03:05:52.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 7326a344-3034-4554-8528-f74e9c1f8600 Correlation ID: 25d78ec5-4c34-41b0-ae55-e68ece8c0e7e Timestamp: 2024-06-29 07:00:15Z
Microsoft.Azure.Commands.Common.Exceptions.AzPSArgumentException: Your Azure credentials have not been set up or have expired, please run Connect-AzAccount to set up your Azure credentials.
A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-06-29T07:00:15.1205639Z, assertion valid from 2024-06-25T02:05:52.0000000Z, expiry time of assertion 2024-06-25T03:05:52.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 7326a344-3034-4554-8528-f74e9c1f8600 Correlation ID: 25d78ec5-4c34-41b0-ae55-e68ece8c0e7e Timestamp: 2024-06-29 07:00:15Z
 ---> MSAL.NetCore.4.56.0.0.MsalServiceException:
	ErrorCode: invalid_client
Microsoft.Identity.Client.MsalServiceException: A configuration issue is preventing authentication - check the error message from the server for details. You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-06-29T07:00:15.1205639Z, assertion valid from 2024-06-25T02:05:52.0000000Z, expiry time of assertion 2024-06-25T03:05:52.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 7326a344-3034-4554-8528-f74e9c1f8600 Correlation ID: 25d78ec5-4c34-41b0-ae55-e68ece8c0e7e Timestamp: 2024-06-29 07:00:15Z
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.ThrowServerException(HttpResponse response, RequestContext requestContext)
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.CreateResponse[T](HttpResponse response, RequestContext requestContext)
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.ExecuteRequestAsync[T](Uri endPoint, HttpMethod method, RequestContext requestContext, Boolean expectErrorsOn200OK, Boolean addCommonHeaders, Func`2 onBeforePostRequestData)
   at Microsoft.Identity.Client.OAuth2.OAuth2Client.GetTokenAsync(Uri endPoint, RequestContext requestContext, Boolean addCommonHeaders, Func`2 onBeforePostRequestHandler)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint, ILoggerAdapter logger)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendHttpAndClearTelemetryAsync(String tokenEndpoint, ILoggerAdapter logger)
   at Microsoft.Identity.Client.OAuth2.TokenClient.SendTokenRequestAsync(IDictionary`2 additionalBodyParameters, String scopeOverride, String tokenEndpointOverride, CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.SendTokenRequestAsync(IDictionary`2 additionalBodyParameters, CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.GetAccessTokenAsync(CancellationToken cancellationToken, ILoggerAdapter logger)
   at Microsoft.Identity.Client.Internal.Requests.ClientCredentialRequest.ExecuteAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.Internal.Requests.RequestBase.RunAsync(CancellationToken cancellationToken)
   at Microsoft.Identity.Client.ApiConfig.Executors.ConfidentialClientExecutor.ExecuteAsync(AcquireTokenCommonParameters commonParameters, AcquireTokenForClientParameters clientParameters, CancellationToken cancellationToken)
   at Microsoft.Azure.PowerShell.Authenticators.Identity.MsalConfidentialClient.AcquireTokenForClientCoreAsync(String[] scopes, String tenantId, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Azure.PowerShell.Authenticators.Identity.MsalConfidentialClient.AcquireTokenForClientAsync(String[] scopes, String tenantId, Boolean async, CancellationToken cancellationToken)
   at Microsoft.Azure.PowerShell.Authenticators.Identity.ClientAssertionCredential.GetTokenAsync(TokenRequestContext requestContext, CancellationToken cancellationToken)
   at Microsoft.Azure.PowerShell.Authenticators.MsalAccessToken.GetAccessTokenAsync(String callerClassName, String parametersLog, TokenCredential tokenCredential, TokenRequestContext requestContext, CancellationToken cancellationToken, String tenantId, String userId, String homeAccountId)
   at Microsoft.Azure.Commands.Common.Authentication.Factories.AuthenticationFactory.Authenticate(IAzureAccount account, IAzureEnvironment environment, String tenant, SecureString password, String promptBehavior, Action`1 promptAction, IAzureTokenCache tokenCache, String resourceId)
   at Microsoft.Azure.Commands.Common.Authentication.Factories.AuthenticationFactory.GetServiceClientCredentials(IAzureContext context, String targetEndpoint, String resourceId)
	StatusCode: 401
	ResponseBody: {"error":"invalid_client","error_description":"AADSTS700024: Client assertion is not within its valid time range. Current time: 2024-06-29T07:00:15.1205639Z, assertion valid from 2024-06-25T02:05:52.0000000Z, expiry time of assertion 2024-06-25T03:05:52.0000000Z. Review the documentation at https://docs.microsoft.com/azure/active-directory/develop/active-directory-certificate-credentials . Trace ID: 7326a344-3034-4554-8528-f74e9c1f8600 Correlation ID: 25d78ec5-4c34-41b0-ae55-e68ece8c0e7e Timestamp: 2024-06-29 07:00:15Z","error_codes":[700024],"timestamp":"2024-06-29 07:00:15Z","trace_id":"7326a344-3034-4554-8528-f74e9c1f8600","correlation_id":"25d78ec5-4c34-41b0-ae55-e68ece8c0e7e","error_uri":"https://login.microsoftonline.com/error?code=700024"}
	Headers: Cache-Control: no-store, no-cache
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
client-request-id: 25d78ec5-4c34-41b0-ae55-e68ece8c0e7e
x-ms-request-id: 7326a344-3034-4554-8528-f74e9c1f8600
x-ms-ests-server: 2.1.18348.7 - WEULR1 ProdSlices
x-ms-clitelem: 1,700024,0,,
x-ms-srs: 1.P
X-XSS-Protection: 0
Set-Cookie: fpc=AhM9E1f8uypNrHWvQ6gp9TqTS9noAQAAAP6nEd4OAAAA; expires=Mon, 29-Jul-2024 07:00:15 GMT; path=/; secure; HttpOnly; SameSite=None, x-ms-gateway-slice=estsfd; path=/; secure; httponly, stsservicecookie=estsfd; path=/; secure; httponly
Date: Sat, 29 Jun 2024 07:00:14 GMT
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Commands.Common.Authentication.Factories.AuthenticationFactory.GetServiceClientCredentials(IAzureContext context, String targetEndpoint, String resourceId)
   at Microsoft.Azure.Commands.KeyVault.Models.KeyVaultDataServiceClient..ctor(IAuthenticationFactory authFactory, IAzureContext context)
   at Microsoft.Azure.Commands.KeyVault.Models.KeyVaultCmdletBase.get_DataServiceClient()
   at Microsoft.Azure.Commands.KeyVault.GetAzureKeyVaultSecret.ExecuteCmdlet()
   at Microsoft.WindowsAzure.Commands.Utilities.Common.AzurePSCmdlet.ProcessRecord()

Thank you in advance for your help.