Troubleshoot ASP.NET Core on Azure App Service and IIS

By Luke Latham and Justin Kotalik

This article provides information on common app startup errors and instructions on how to diagnose errors when an app is deployed to Azure App Service or IIS:

App startup errors
Explains common startup HTTP status code scenarios.

Troubleshoot on Azure App Service
Provides troubleshooting advice for apps deployed to Azure App Service.

Troubleshoot on IIS
Provides troubleshooting advice for apps deployed to IIS or running on IIS Express locally. The guidance applies to both Windows Server and Windows desktop deployments.

Clear package caches
Explains what to do when incoherent packages break an app when performing major upgrades or changing package versions.

Additional resources
Lists additional troubleshooting topics.

App startup errors

In Visual Studio, an ASP.NET Core project defaults to IIS Express hosting during debugging. A 502.5 - Process Failure or a 500.30 - Start Failure that occurs when debugging locally can be diagnosed using the advice in this topic.

In Visual Studio, an ASP.NET Core project defaults to IIS Express hosting during debugging. A 502.5 Process Failure that occurs when debugging locally can be diagnosed using the advice in this topic.

500 Internal Server Error

The app starts, but an error prevents the server from fulfilling the request.

This error occurs within the app's code during startup or while creating a response. The response may contain no content, or the response may appear as a 500 Internal Server Error in the browser. The Application Event Log usually states that the app started normally. From the server's perspective, that's correct. The app did start, but it can't generate a valid response. Run the app at a command prompt on the server or enable the ASP.NET Core Module stdout log to troubleshoot the problem.

500.0 In-Process Handler Load Failure

The worker process fails. The app doesn't start.

The ASP.NET Core Module fails to find the .NET Core CLR and find the in-process request handler (aspnetcorev2_inprocess.dll). Check that:

500.0 Out-Of-Process Handler Load Failure

The worker process fails. The app doesn't start.

The ASP.NET Core Module fails to find the out-of-process hosting request handler. Make sure the aspnetcorev2_outofprocess.dll is present in a subfolder next to aspnetcorev2.dll.

500.0 In-Process Handler Load Failure

The worker process fails. The app doesn't start.

An unknown error occurred loading ASP.NET Core Module components. Take one of the following actions:

500.30 In-Process Startup Failure

The worker process fails. The app doesn't start.

The ASP.NET Core Module attempts to start the .NET Core CLR in-process, but it fails to start. The cause of a process startup failure can usually be determined from entries in the Application Event Log and the ASP.NET Core Module stdout log.

A common failure condition is the app is misconfigured due to targeting a version of the ASP.NET Core shared framework that isn't present. Check which versions of the ASP.NET Core shared framework are installed on the target machine.

500.31 ANCM Failed to Find Native Dependencies

The worker process fails. The app doesn't start.

The ASP.NET Core Module attempts to start the .NET Core runtime in-process, but it fails to start. The most common cause of this startup failure is when the Microsoft.NETCore.App or Microsoft.AspNetCore.App runtime isn't installed. If the app is deployed to target ASP.NET Core 3.0 and that version doesn't exist on the machine, this error occurs. An example error message follows:

The specified framework 'Microsoft.NETCore.App', version '3.0.0' was not found.
  - The following frameworks were found:
      2.2.1 at [C:\Program Files\dotnet\x64\shared\Microsoft.NETCore.App]
      3.0.0-preview5-27626-15 at [C:\Program Files\dotnet\x64\shared\Microsoft.NETCore.App]
      3.0.0-preview6-27713-13 at [C:\Program Files\dotnet\x64\shared\Microsoft.NETCore.App]
      3.0.0-preview6-27714-15 at [C:\Program Files\dotnet\x64\shared\Microsoft.NETCore.App]
      3.0.0-preview6-27723-08 at [C:\Program Files\dotnet\x64\shared\Microsoft.NETCore.App]

The error message lists all the installed .NET Core versions and the version requested by the app. To fix this error, either:

  • Install the appropriate version of .NET Core on the machine.
  • Change the app to target a version of .NET Core that's present on the machine.
  • Publish the app as a self-contained deployment.

When running in development (the ASPNETCORE_ENVIRONMENT environment variable is set to Development), the specific error is written to the HTTP response. The cause of a process startup failure is also found in the Application Event Log.

500.32 ANCM Failed to Load dll

The worker process fails. The app doesn't start.

The most common cause for this error is that the app is published for an incompatible processor architecture. If the worker process is running as a 32-bit app and the app was published to target 64-bit, this error occurs.

To fix this error, either:

500.33 ANCM Request Handler Load Failure

The worker process fails. The app doesn't start.

The app didn't reference the Microsoft.AspNetCore.App framework. Only apps targeting the Microsoft.AspNetCore.App framework can be hosted by the ASP.NET Core Module.

To fix this error, confirm that the app is targeting the Microsoft.AspNetCore.App framework. Check the .runtimeconfig.json to verify the framework targeted by the app.

500.34 ANCM Mixed Hosting Models Not Supported

The worker process can't run both an in-process app and an out-of-process app in the same process.

To fix this error, run apps in separate IIS application pools.

500.35 ANCM Multiple In-Process Applications in same Process

The worker process can't run both an in-process app and an out-of-process app in the same process.

To fix this error, run apps in separate IIS application pools.

500.36 ANCM Out-Of-Process Handler Load Failure

The out-of-process request handler, aspnetcorev2_outofprocess.dll, isn't next to the aspnetcorev2.dll file. This indicates a corrupted installation of the ASP.NET Core Module.

To fix this error, repair the installation of the .NET Core Hosting Bundle (for IIS) or Visual Studio (for IIS Express).

500.37 ANCM Failed to Start Within Startup Time Limit

ANCM failed to start within the provied startup time limit. By default, the timeout is 120 seconds.

This error can occur when starting a large number of apps on the same machine. Check for CPU/Memory usage spikes on the server during startup. You may need to stagger the startup process of multiple apps.

502.5 Process Failure

The worker process fails. The app doesn't start.

The ASP.NET Core Module attempts to start the worker process but it fails to start. The cause of a process startup failure can usually be determined from entries in the Application Event Log and the ASP.NET Core Module stdout log.

A common failure condition is the app is misconfigured due to targeting a version of the ASP.NET Core shared framework that isn't present. Check which versions of the ASP.NET Core shared framework are installed on the target machine. The shared framework is the set of assemblies (.dll files) that are installed on the machine and referenced by a metapackage such as Microsoft.AspNetCore.App. The metapackage reference can specify a minimum required version. For more information, see The shared framework.

The 502.5 Process Failure error page is returned when a hosting or app misconfiguration causes the worker process to fail:

Failed to start application (ErrorCode '0x800700c1')

EventID: 1010
Source: IIS AspNetCore Module V2
Failed to start application '/LM/W3SVC/6/ROOT/', ErrorCode '0x800700c1'.

The app failed to start because the app's assembly (.dll) couldn't be loaded.

This error occurs when there's a bitness mismatch between the published app and the w3wp/iisexpress process.

Confirm that the app pool's 32-bit setting is correct:

  1. Select the app pool in IIS Manager's Application Pools.
  2. Select Advanced Settings under Edit Application Pool in the Actions panel.
  3. Set Enable 32-Bit Applications:
    • If deploying a 32-bit (x86) app, set the value to True.
    • If deploying a 64-bit (x64) app, set the value to False.

Connection reset

If an error occurs after the headers are sent, it's too late for the server to send a 500 Internal Server Error when an error occurs. This often happens when an error occurs during the serialization of complex objects for a response. This type of error appears as a connection reset error on the client. Application logging can help troubleshoot these types of errors.

Default startup limits

The ASP.NET Core Module is configured with a default startupTimeLimit of 120 seconds. When left at the default value, an app may take up to two minutes to start before the module logs a process failure. For information on configuring the module, see Attributes of the aspNetCore element.

Troubleshoot on Azure App Service

Important

ASP.NET Core preview releases with Azure App Service

ASP.NET Core preview releases aren't deployed to Azure App Service by default. To host an app that uses an ASP.NET Core preview release, see Deploy ASP.NET Core preview release to Azure App Service.

Application Event Log (Azure App Service)

To access the Application Event Log, use the Diagnose and solve problems blade in the Azure portal:

  1. In the Azure portal, open the app in App Services.
  2. Select Diagnose and solve problems.
  3. Select the Diagnostic Tools heading.
  4. Under Support Tools, select the Application Events button.
  5. Examine the latest error provided by the IIS AspNetCoreModule or IIS AspNetCoreModule V2 entry in the Source column.

An alternative to using the Diagnose and solve problems blade is to examine the Application Event Log file directly using Kudu:

  1. Open Advanced Tools in the Development Tools area. Select the Go→ button. The Kudu console opens in a new browser tab or window.
  2. Using the navigation bar at the top of the page, open Debug console and select CMD.
  3. Open the LogFiles folder.
  4. Select the pencil icon next to the eventlog.xml file.
  5. Examine the log. Scroll to the bottom of the log to see the most recent events.

Run the app in the Kudu console

Many startup errors don't produce useful information in the Application Event Log. You can run the app in the Kudu Remote Execution Console to discover the error:

  1. Open Advanced Tools in the Development Tools area. Select the Go→ button. The Kudu console opens in a new browser tab or window.
  2. Using the navigation bar at the top of the page, open Debug console and select CMD.

Test a 32-bit (x86) app

Current release

  1. cd d:\home\site\wwwroot
  2. Run the app:

The console output from the app, showing any errors, is piped to the Kudu console.

Framework-dependent deployment running on a preview release

Requires installing the ASP.NET Core {VERSION} (x86) Runtime site extension.

  1. cd D:\home\SiteExtensions\AspNetCoreRuntime.{X.Y}.x32 ({X.Y} is the runtime version)
  2. Run the app: dotnet \home\site\wwwroot\{ASSEMBLY NAME}.dll

The console output from the app, showing any errors, is piped to the Kudu console.

Test a 64-bit (x64) app

Current release

The console output from the app, showing any errors, is piped to the Kudu console.

Framework-dependent deployment running on a preview release

Requires installing the ASP.NET Core {VERSION} (x64) Runtime site extension.

  1. cd D:\home\SiteExtensions\AspNetCoreRuntime.{X.Y}.x64 ({X.Y} is the runtime version)
  2. Run the app: dotnet \home\site\wwwroot\{ASSEMBLY NAME}.dll

The console output from the app, showing any errors, is piped to the Kudu console.

ASP.NET Core Module stdout log (Azure App Service)

The ASP.NET Core Module stdout log often records useful error messages not found in the Application Event Log. To enable and view stdout logs:

  1. Navigate to the Diagnose and solve problems blade in the Azure portal.
  2. Under SELECT PROBLEM CATEGORY, select the Web App Down button.
  3. Under Suggested Solutions > Enable Stdout Log Redirection, select the button to Open Kudu Console to edit Web.Config.
  4. In the Kudu Diagnostic Console, open the folders to the path site > wwwroot. Scroll down to reveal the web.config file at the bottom of the list.
  5. Click the pencil icon next to the web.config file.
  6. Set stdoutLogEnabled to true and change the stdoutLogFile path to: \\?\%home%\LogFiles\stdout.
  7. Select Save to save the updated web.config file.
  8. Make a request to the app.
  9. Return to the Azure portal. Select the Advanced Tools blade in the DEVELOPMENT TOOLS area. Select the Go→ button. The Kudu console opens in a new browser tab or window.
  10. Using the navigation bar at the top of the page, open Debug console and select CMD.
  11. Select the LogFiles folder.
  12. Inspect the Modified column and select the pencil icon to edit the stdout log with the latest modification date.
  13. When the log file opens, the error is displayed.

Disable stdout logging when troubleshooting is complete:

  1. In the Kudu Diagnostic Console, return to the path site > wwwroot to reveal the web.config file. Open the web.config file again by selecting the pencil icon.
  2. Set stdoutLogEnabled to false.
  3. Select Save to save the file.

For more information, see ASP.NET Core Module.

Warning

Failure to disable the stdout log can lead to app or server failure. There's no limit on log file size or the number of log files created. Only use stdout logging to troubleshoot app startup problems.

For general logging in an ASP.NET Core app after startup, use a logging library that limits log file size and rotates logs. For more information, see third-party logging providers.

ASP.NET Core Module debug log (Azure App Service)

The ASP.NET Core Module debug log provides additional, deeper logging from the ASP.NET Core Module. To enable and view stdout logs:

  1. To enable the enhanced diagnostic log, perform either of the following:
    • Follow the instructions in Enhanced diagnostic logs to configure the app for an enhanced diagnostic logging. Redeploy the app.
    • Add the <handlerSettings> shown in Enhanced diagnostic logs to the live app's web.config file using the Kudu console:
      1. Open Advanced Tools in the Development Tools area. Select the Go→ button. The Kudu console opens in a new browser tab or window.
      2. Using the navigation bar at the top of the page, open Debug console and select CMD.
      3. Open the folders to the path site > wwwroot. Edit the web.config file by selecting the pencil button. Add the <handlerSettings> section as shown in Enhanced diagnostic logs. Select the Save button.
  2. Open Advanced Tools in the Development Tools area. Select the Go→ button. The Kudu console opens in a new browser tab or window.
  3. Using the navigation bar at the top of the page, open Debug console and select CMD.
  4. Open the folders to the path site > wwwroot. If you didn't supply a path for the aspnetcore-debug.log file, the file appears in the list. If you supplied a path, navigate to the location of the log file.
  5. Open the log file with the pencil button next to the file name.

Disable debug logging when troubleshooting is complete:

To disable the enhanced debug log, perform either of the following:

  • Remove the <handlerSettings> from the web.config file locally and redeploy the app.
  • Use the Kudu console to edit the web.config file and remove the <handlerSettings> section. Save the file.

For more information, see ASP.NET Core Module.

Warning

Failure to disable the debug log can lead to app or server failure. There's no limit on log file size. Only use debug logging to troubleshoot app startup problems.

For general logging in an ASP.NET Core app after startup, use a logging library that limits log file size and rotates logs. For more information, see third-party logging providers.

Slow or hanging app (Azure App Service)

When an app responds slowly or hangs on a request, see the following articles:

Monitoring blades

Monitoring blades provide an alternative troubleshooting experience to the methods described earlier in the topic. These blades can be used to diagnose 500-series errors.

Confirm that the ASP.NET Core Extensions are installed. If the extensions aren't installed, install them manually:

  1. In the DEVELOPMENT TOOLS blade section, select the Extensions blade.
  2. The ASP.NET Core Extensions should appear in the list.
  3. If the extensions aren't installed, select the Add button.
  4. Choose the ASP.NET Core Extensions from the list.
  5. Select OK to accept the legal terms.
  6. Select OK on the Add extension blade.
  7. An informational pop-up message indicates when the extensions are successfully installed.

If stdout logging isn't enabled, follow these steps:

  1. In the Azure portal, select the Advanced Tools blade in the DEVELOPMENT TOOLS area. Select the Go→ button. The Kudu console opens in a new browser tab or window.
  2. Using the navigation bar at the top of the page, open Debug console and select CMD.
  3. Open the folders to the path site > wwwroot and scroll down to reveal the web.config file at the bottom of the list.
  4. Click the pencil icon next to the web.config file.
  5. Set stdoutLogEnabled to true and change the stdoutLogFile path to: \\?\%home%\LogFiles\stdout.
  6. Select Save to save the updated web.config file.

Proceed to activate diagnostic logging:

  1. In the Azure portal, select the Diagnostics logs blade.
  2. Select the On switch for Application Logging (Filesystem) and Detailed error messages. Select the Save button at the top of the blade.
  3. To include failed request tracing, also known as Failed Request Event Buffering (FREB) logging, select the On switch for Failed request tracing.
  4. Select the Log stream blade, which is listed immediately under the Diagnostics logs blade in the portal.
  5. Make a request to the app.
  6. Within the log stream data, the cause of the error is indicated.

Be sure to disable stdout logging when troubleshooting is complete.

To view the failed request tracing logs (FREB logs):

  1. Navigate to the Diagnose and solve problems blade in the Azure portal.
  2. Select Failed Request Tracing Logs from the SUPPORT TOOLS area of the sidebar.

See Failed request traces section of the Enable diagnostics logging for web apps in Azure App Service topic and the Application performance FAQs for Web Apps in Azure: How do I turn on failed request tracing? for more information.

For more information, see Enable diagnostics logging for web apps in Azure App Service.

Warning

Failure to disable the stdout log can lead to app or server failure. There's no limit on log file size or the number of log files created.

For routine logging in an ASP.NET Core app, use a logging library that limits log file size and rotates logs. For more information, see third-party logging providers.

Troubleshoot on IIS

Application Event Log (IIS)

Access the Application Event Log:

  1. Open the Start menu, search for Event Viewer, and then select the Event Viewer app.
  2. In Event Viewer, open the Windows Logs node.
  3. Select Application to open the Application Event Log.
  4. Search for errors associated with the failing app. Errors have a value of IIS AspNetCore Module or IIS Express AspNetCore Module in the Source column.

Run the app at a command prompt

Many startup errors don't produce useful information in the Application Event Log. You can find the cause of some errors by running the app at a command prompt on the hosting system.

Framework-dependent deployment

If the app is a framework-dependent deployment:

  1. At a command prompt, navigate to the deployment folder and run the app by executing the app's assembly with dotnet.exe. In the following command, substitute the name of the app's assembly for <assembly_name>: dotnet .\<assembly_name>.dll.
  2. The console output from the app, showing any errors, is written to the console window.
  3. If the errors occur when making a request to the app, make a request to the host and port where Kestrel listens. Using the default host and post, make a request to http://localhost:5000/. If the app responds normally at the Kestrel endpoint address, the problem is more likely related to the hosting configuration and less likely within the app.

Self-contained deployment

If the app is a self-contained deployment:

  1. At a command prompt, navigate to the deployment folder and run the app's executable. In the following command, substitute the name of the app's assembly for <assembly_name>: <assembly_name>.exe.
  2. The console output from the app, showing any errors, is written to the console window.
  3. If the errors occur when making a request to the app, make a request to the host and port where Kestrel listens. Using the default host and post, make a request to http://localhost:5000/. If the app responds normally at the Kestrel endpoint address, the problem is more likely related to the hosting configuration and less likely within the app.

ASP.NET Core Module stdout log (IIS)

To enable and view stdout logs:

  1. Navigate to the site's deployment folder on the hosting system.
  2. If the logs folder isn't present, create the folder. For instructions on how to enable MSBuild to create the logs folder in the deployment automatically, see the Directory structure topic.
  3. Edit the web.config file. Set stdoutLogEnabled to true and change the stdoutLogFile path to point to the logs folder (for example, .\logs\stdout). stdout in the path is the log file name prefix. A timestamp, process id, and file extension are added automatically when the log is created. Using stdout as the file name prefix, a typical log file is named stdout_20180205184032_5412.log.
  4. Ensure your application pool's identity has write permissions to the logs folder.
  5. Save the updated web.config file.
  6. Make a request to the app.
  7. Navigate to the logs folder. Find and open the most recent stdout log.
  8. Study the log for errors.

Disable stdout logging when troubleshooting is complete:

  1. Edit the web.config file.
  2. Set stdoutLogEnabled to false.
  3. Save the file.

For more information, see ASP.NET Core Module.

Warning

Failure to disable the stdout log can lead to app or server failure. There's no limit on log file size or the number of log files created.

For routine logging in an ASP.NET Core app, use a logging library that limits log file size and rotates logs. For more information, see third-party logging providers.

ASP.NET Core Module debug log (IIS)

Add the following handler settings to the app's web.config file to enable ASP.NET Core Module debug log:

<aspNetCore ...>
  <handlerSettings>
    <handlerSetting name="debugLevel" value="file" />
    <handlerSetting name="debugFile" value="c:\temp\ancm.log" />
  </handlerSettings>
</aspNetCore>

Confirm that the path specified for the log exists and that the app pool's identity has write permissions to the location.

For more information, see ASP.NET Core Module.

Enable the Developer Exception Page

The ASPNETCORE_ENVIRONMENT environment variable can be added to web.config to run the app in the Development environment. As long as the environment isn't overridden in app startup by UseEnvironment on the host builder, setting the environment variable allows the Developer Exception Page to appear when the app is run.

<aspNetCore processPath="dotnet"
      arguments=".\MyApp.dll"
      stdoutLogEnabled="false"
      stdoutLogFile=".\logs\stdout"
      hostingModel="InProcess">
  <environmentVariables>
    <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Development" />
  </environmentVariables>
</aspNetCore>
<aspNetCore processPath="dotnet"
      arguments=".\MyApp.dll"
      stdoutLogEnabled="false"
      stdoutLogFile=".\logs\stdout">
  <environmentVariables>
    <environmentVariable name="ASPNETCORE_ENVIRONMENT" value="Development" />
  </environmentVariables>
</aspNetCore>

Setting the environment variable for ASPNETCORE_ENVIRONMENT is only recommended for use on staging and testing servers that aren't exposed to the Internet. Remove the environment variable from the web.config file after troubleshooting. For information on setting environment variables in web.config, see environmentVariables child element of aspNetCore.

Obtain data from an app

If an app is capable of responding to requests, obtain request, connection, and additional data from the app using terminal inline middleware. For more information and sample code, see Troubleshoot ASP.NET Core projects.

Slow or hanging app (IIS)

A crash dump is a snapshot of the system's memory and can help determine the cause of an app crash, startup failure, or slow app.

App crashes or encounters an exception

Obtain and analyze a dump from Windows Error Reporting (WER):

  1. Create a folder to hold crash dump files at c:\dumps. The app pool must have write access to the folder.

  2. Run the EnableDumps PowerShell script:

  3. Run the app under the conditions that cause the crash to occur.

  4. After the crash has occurred, run the DisableDumps PowerShell script:

After an app crashes and dump collection is complete, the app is allowed to terminate normally. The PowerShell script configures WER to collect up to five dumps per app.

Warning

Crash dumps might take up a large amount of disk space (up to several gigabytes each).

App hangs, fails during startup, or runs normally

When an app hangs (stops responding but doesn't crash), fails during startup, or runs normally, see User-Mode Dump Files: Choosing the Best Tool to select an appropriate tool to produce the dump.

Analyze the dump

A dump can be analyzed using several approaches. For more information, see Analyzing a User-Mode Dump File.

Clear package caches

Sometimes a functioning app fails immediately after upgrading either the .NET Core SDK on the development machine or changing package versions within the app. In some cases, incoherent packages may break an app when performing major upgrades. Most of these issues can be fixed by following these instructions:

  1. Delete the bin and obj folders.

  2. Clear the package caches by executing dotnet nuget locals all --clear from a command shell.

    Clearing package caches can also be accomplished with the nuget.exe tool and executing the command nuget locals all -clear. nuget.exe isn't a bundled install with the Windows desktop operating system and must be obtained separately from the NuGet website.

  3. Restore and rebuild the project.

  4. Delete all of the files in the deployment folder on the server prior to redeploying the app.

Additional resources

Azure documentation

Visual Studio documentation

Visual Studio Code documentation