IIS: Failed to load CoreCLR with non-existing file

Müller Matthias 0

Hello there

We have really strange behavior; having an IIS, which hosts multiple .NET Core (respectively .NET7 and .NET8) systems, we added a new one.

Since then, we very randomly receive errors like this for seemingly all the .NET 8 systems:

Application '/LM/W3SVC/43/ROOT' with physical root 'C:\inetpub\wwwroot\xxx_dev4' failed to load coreclr. Exception message: Unexpected exception: directory_iterator::directory_iterator: The system cannot find the path specified.: "C:\inetpub\wwwroot\xx_dev4\wwwroot\bootstrap"

Now, the problem seems to be, that for the .NET8 applications, we used the inprocess hosting model, not the out of process one like here explained: https://stackoverflow.com/questions/59369322/net-core-3-0-and-iis-http-error-500-30-ancm-in-process-start-failure-failed

What bothers us: We used inprocess one for almost a year now and never had problems. Even systems, which were not touched for months, started to get this error after a new system was deployed. Just to explain: the wwwroot\bootstrap folder doesn't nor should exist for the systems failing, and sometimes it gives error for other files.

Restarting the IIS also randomly threw this errors for certain paths, almost like a first-system-wins pattern.

This opens a whole can of problems; we can destabilize existing systems just by using inprocess models, even when not touching them. If eels like the IIS is mixing up requests and systems, but I have no idea how to further analyze the issue, as it's not always reproducible.

Do you have any idea, how something like this can happen?

Greetings

Matthias

Yurong Dai-MSFT 2,781 Reputation points Microsoft Vendor

2024-03-21T08:29:18.01+00:00

Hi @Müller Matthias,
What error status code did you get? Please provide as much detailed error information as possible.
Müller Matthias 0 Reputation points

2024-03-21T09:26:38.3633333+00:00
Hello thank you for the feedback. Unfortunately the only source for the error is the event viewer. This is the XML for the error:

<Event xmlns="__http://schemas.microsoft.com/win/2004/08/events/event__">

<System>

<Provider Name="IIS AspNetCore Module V2" />

<EventID Qualifiers="0">1007</EventID>

<Version>0</Version>

<Level>2</Level>

<Task>0</Task>

<Opcode>0</Opcode>

<Keywords>0x80000000000000</Keywords>

<TimeCreated SystemTime="2024-03-20T15:55:16.4971231Z" />

<EventRecordID>81637</EventRecordID>

<Correlation />

<Execution ProcessID="0" ThreadID="0" />

<Channel>Application</Channel>

<Computer>WEB-TEST.server.local</Computer>

<Security />

</System>

<EventData>

<Data>Application '/LM/W3SVC/44/ROOT' with physical root 'C:\inetpub\wwwroot\system_dev2' failed to load coreclr. Exception message: Unexpected exception: directory_iterator::directory_iterator: The system cannot find the path specified.: "C:\inetpub\wwwroot\system_dev2\runtimes\browser-wasm"</Data>

<Data>Process Id: 11332.</Data>

<Data>File Version: 18.0.24047.3. Description: IIS ASP.NET Core Module V2 Request Handler. Commit:</Data>

</EventData>

</Event>

As said, this file is not supposed to be there, nor should it be.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Müller Matthias 0 Reputation points

2024-03-22T08:03:18.1966667+00:00

@Yurong Dai-MSFT Thank you for the feedback. Probably the randomness doesn't actually matter too much. The question is more: When multiple .NET 8 systems run with inprocess hosting model, is it known that IIS mixes up something? Like threads, Routes, or the likes?
Yurong Dai-MSFT 2,781 Reputation points Microsoft Vendor

2024-03-22T09:36:11.24+00:00

Hi @Müller Matthias,
In the in-process hosting model, ASP.NET Core applications run directly in IIS's worker process. When multiple .NET Core applications run in the in-process hosting model, they share the same worker process. This results in HTTP error 500.35 - ANCM Multiple In-Process Applications in same Process, because the worker process can't run multiple in-process apps in the same process.

To resolve this issue, run the applications in separate IIS application pools, or use the out-of-process hosting model to ensure that each application runs in its own process.

https://learn.microsoft.com/en-us/aspnet/core/test/troubleshoot-azure-iis?view=aspnetcore-8.0#50035-ancm-multiple-in-process-applications-in-same-process
Müller Matthias 0 Reputation points

2024-03-22T09:39:17.0933333+00:00

@Yurong Dai-MSFT We had (and still have) this exact combination but we had separate application Pools. So, inprocess + seperate App Pools seems still to influence the threading somewhere in IIS. Can it be due to the https://learn.microsoft.com/de-de/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-8.0?
Yurong Dai-MSFT 2,781 Reputation points Microsoft Vendor

2024-03-27T09:27:59.51+00:00

It is recommended that you use the out-of-process hosting model. In this model, each ASP.NET Core application runs in its own process entirely separate from IIS. This completely eliminates any potential conflicts related to sharing resources within the IIS worker process.
Müller Matthias 0 Reputation points

2024-03-27T11:28:23.36+00:00

@Yurong Dai-MSFT Thank you for the feedback. So there is a known issue that the IIS work processes shares resources, although having different app pools?
Yurong Dai-MSFT 2,781 Reputation points Microsoft Vendor

2024-04-01T09:41:08.7366667+00:00

In IIS, each application pool runs in its own worker process, which means that different application pools are isolated from each other and have their own independent memory space and resources. Therefore, in theory, different application pools should not share resources. However, resource sharing or competition may occur under certain circumstances. I can only explain so much.
Müller Matthias 0 Reputation points

2024-04-02T07:26:00.9766667+00:00

@Yurong Gao I guess you don't have by chance a link about this circumstances?