Collect diagnostics in containers

The same diagnostics tools that are useful for diagnosing .NET Core issues in other scenarios also work in Docker containers. However, some of the tools require special steps to work in a container. This article covers how tools for gathering performance traces and collecting dumps can be used in Docker containers.

Using .NET Core CLI tools in a container

These tools apply to: ✔️ .NET Core 3.0 SDK and later versions

The .NET Core global CLI diagnostic tools (dotnet-counters, dotnet-dump, dotnet-gcdump, and dotnet-trace) are designed to work in a wide variety of environments and should all work directly in Docker containers. Because of this, these tools are the preferred method of collecting diagnostic information for .NET Core scenarios targeting .NET Core 3.0 or above (or 3.1 or above in the case of dotnet-gcdump) in containers.

The only complicating factor of using these tools in a container is that they are installed with the .NET Core SDK and many Docker containers run without the .NET Core SDK present. One easy solution to this problem is to install the tools in the initial Docker image. The tools don't need the .NET Core SDK to run, only to be installed. Therefore, it's possible to create a Dockerfile with a multi-stage build that installs the tools in a build stage (where the .NET Core SDK is present) and then copies the binaries into the final image. The only downside to this approach is increased Docker image size.

# In build stage
# Install desired .NET CLI diagnostics tools
RUN dotnet tool install --tool-path /tools dotnet-trace
RUN dotnet tool install --tool-path /tools dotnet-counters
RUN dotnet tool install --tool-path /tools dotnet-dump

...

# In final stage
# Copy diagnostics tools
WORKDIR /tools
COPY --from=build /tools .

Alternatively, the .NET Core SDK can be installed in a container when needed in order to install the CLI tools. Be aware that installing the .NET Core SDK will have the side-effect of reinstalling the .NET Core runtime. So be sure to install the version of the SDK that matches the runtime present in the container.

Using .NET Core global CLI tools in a sidecar container

If you would like to use .NET Core global CLI diagnostic tools to diagnose processes in a different container, bear the following additional requirements in mind:

  1. The containers must share a process namespace (so that tools in the sidecar container can access processes in the target container).
  2. The .NET Core global CLI diagnostic tools need access to files the .NET Core runtime writes to the /tmp directory, so the /tmp directory must be shared between the target and sidecar container via a volume mount. This could be done, for example, by having the containers share a common volume or a Kubernetes emptyDir volume. If you attempt to use the diagnostic tools from a sidecar container without sharing the /tmp directory, you will get an error about the process "not running compatible .NET runtime."

Using PerfCollect in a container

This tool applies to: ✔️ .NET Core 2.1 and later versions

The PerfCollect script is useful for collecting performance traces and is the recommended tool for collecting traces prior to .NET Core 3.0. If using PerfCollect in a container, keep the following requirements in mind:

  • PerfCollect requires the SYS_ADMIN capability (in order to run the perf tool), so be sure the container is started with that capability.

  • PerfCollect requires some environment variables be set prior to the app it is profiling starting. These can be set either in a Dockerfile or when starting the container. Because these variables shouldn't be set in normal production environments, it's common to just add them when starting a container that will be profiled. The two variables that PerfCollect requires are:

    • DOTNET_PerfMapEnabled=1
    • DOTNET_EnableEventLog=1

    Note

    .NET 6 standardizes on the prefix DOTNET_ instead of COMPlus_ for environment variables that configure .NET run-time behavior. However, the COMPlus_ prefix will continue to work. If you're using a previous version of the .NET runtime, you should still use the COMPlus_ prefix for environment variables.

Using PerfCollect in a sidecar container

If you would like to run PerfCollect in one container to profile a .NET Core process in a different container, the experience is almost the same except for these differences:

  • The environment variables mentioned previously (DOTNET_PerfMapEnabled and DOTNET_EnableEventLog) must be set for the target container (not the one running PerfCollect).
  • The container running PerfCollect must have the SYS_ADMIN capability (not the target container).
  • The two containers must share a process namespace.

Using createdump in a container

This tool applies to: ✔️ .NET Core 2.1 and later versions

An alternative to dotnet-dump, createdump can be used for creating core dumps on Linux containing both native and managed information. The createdump tool is installed with the .NET Core runtime and can be found next to libcoreclr.so (typically in "/usr/share/dotnet/shared/Microsoft.NETCore.App/[version]"). The tool works the same in a container as it does in non-containerized Linux environments with the single exception that the tool requires the SYS_PTRACE capability, so the Docker container must be started with that capability.

Using createdump in a sidecar container

If you would like to use createdump to create a dump from a process in a different container, the experience is almost the same except for these differences:

  • The container running createdump must have the SYS_PTRACE capability (not the target container).
  • The two containers must share a process namespace.