IIS site will suddenly not start

PowerShell Universal
1

Hi guys

After coming back from the weekend suddenly our IIS PoshUD site will not start after no changes and running fine for several months.

It’s throwing the below errors when we try to start the site in IIS.

Troubleshooting attempted:

  • Ensured no changes to IIS, the server itself, or the application directories (all timestamps are old)
  • Rebooted server
  • Ensured no recent patches applied to IIS or Windows
  • All other IIS sites are working normally but these are unrelated to PoshUD
  • Ran unblock-file across the directory

Any assistance would be greatly appreciated.

From event log:

Application '/LM/W3SVC/8/ROOT' with physical root 'C:\inetpub\PoshUD\' failed to start process with commandline 'c:\inetpub\poshud\Universal.Server.exe ' with multiple retries. Failed to bind to port '33931'. First 30KB characters of captured stdout and stderr logs from multiple retries:

From ProgramData logs:

2022-07-13T10:43:26.6541597+09:30  [INF] User profile is available. Using '"C:\Users\username\AppData\Local\ASP.NET\DataProtection-Keys"' as key repository and Windows DPAPI to encrypt keys at rest. (7ac5e29c)

From application folder logs:

2022-07-13 10:41:26 [INFO]  (Hangfire.BackgroundJobServer) Starting Hangfire Server using job storage: 'Hangfire.MemoryStorage.MemoryStorage'
2022-07-13 10:41:26 [INFO]  (Hangfire.BackgroundJobServer) Using the following options for Hangfire Server:
    Worker count: 10
    Listening queues: 'default'
    Shutdown timeout: 00:00:15
    Schedule polling interval: 00:00:15
2022-07-13 10:41:26 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server aws-syd-web01:8512:2b01a197 successfully announced in 153.073 ms
2022-07-13 10:41:26 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server aws-syd-web01:8512:2b01a197 is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, ExpirationManager, CountersAggregator, Worker, DelayedJobScheduler, RecurringJobScheduler...
2022-07-13 10:41:26 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server aws-syd-web01:8512:2b01a197 all the dispatchers started
2022-07-13 10:43:26 [INFO]  (Hangfire.BackgroundJobServer) Starting Hangfire Server using job storage: 'Hangfire.MemoryStorage.MemoryStorage'
2022-07-13 10:43:26 [INFO]  (Hangfire.BackgroundJobServer) Using the following options for Hangfire Server:
    Worker count: 10
    Listening queues: 'default'
    Shutdown timeout: 00:00:15
    Schedule polling interval: 00:00:15
2022-07-13 10:43:26 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server aws-syd-web01:936:dd06e913 successfully announced in 150.0911 ms
2022-07-13 10:43:26 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server aws-syd-web01:936:dd06e913 is starting the registered dispatchers: ServerWatchdog, ServerJobCancellationWatcher, ExpirationManager, CountersAggregator, Worker, DelayedJobScheduler, RecurringJobScheduler...
2022-07-13 10:43:27 [INFO]  (Hangfire.Server.BackgroundServerProcess) Server aws-syd-web01:936:dd06e913 all the dispatchers started
2

This is where I would start. Do you have another site on the server, or another service/process that has hijacked port 33931? While the PSU site is stopped, see if port 33931 is still listening when it shouldn’t be.

Get-NetTCPConnection -LocalPort 33931
3

Hey @OtherPeoplesCode that was my first thought too but the error appears to be a bit of a furphy - the ports its mentioning are definitely not in use.

4

I’ve collected additional logs by increasing the debug level. @Adam do you have any ideas?

[aspnetcorev2.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\aspnetcorev2.dll'. Process Id: 6796.. File Version: 13.1.20074.3. Description: IIS ASP.NET Core Module V2. Commit: e81033e094d4663ffd227bb4aed30b76b0631e6d.
[aspnetcorev2.dll] Loading request handler:  'C:\Program Files\IIS\Asp.Net Core Module\V2\13.1.20074\aspnetcorev2_outofprocess.dll'
[aspnetcorev2.dll] Creating handler application
[aspnetcorev2_outofprocess.dll] Initializing logs for 'C:\Program Files\IIS\Asp.Net Core Module\V2\13.1.20074\aspnetcorev2_outofprocess.dll'. Process Id: 6796.. File Version: 13.1.20074.3. Description: IIS ASP.NET Core Module V2 Request Handler. Commit: e81033e094d4663ffd227bb4aed30b76b0631e6d.
[aspnetcorev2_outofprocess.dll] Starting app_offline monitoring in application 'C:\inetpub\PoshUD\'
[aspnetcorev2_outofprocess.dll] Starting file watcher thread
[aspnetcorev2_outofprocess.dll] FORWARDING_HANDLER::FORWARDING_HANDLER
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x8027025a at F:\workspace\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\OutOfProcessRequestHandler\serverprocess.cpp:726 
[aspnetcorev2_outofprocess.dll] Failed HRESULT returned: 0x8027025a at F:\workspace\_work\1\s\src\Servers\IIS\AspNetCoreModuleV2\OutOfProcessRequestHandler\serverprocess.cpp:897 
[aspnetcorev2_outofprocess.dll] Event Log: 'Application '/LM/W3SVC/8/ROOT' with physical root 'C:\inetpub\PoshUD\' failed to start process with commandline 'c:\inetpub\poshud\Universal.Server.exe ' at stage 'PostStartCheck', ErrorCode = '0x8027025a', assigned port 36158, retryCounter '1'.' 
End Event Log Message.
5

It was a licensing issue. Ran procmon and saw lots of traffic to store.ironmansoftware.com.

Deleted database and license files.

Restarted server, reapplied licenses.