Edit

Implementing resource controls for Windows containers

  • Article

Applies to: Windows Server 2022, Windows Server 2019

There are several resource controls that can be implemented on a per-container and per-resource basis. By default, containers run are subject to typical Windows resource management, which in general is fair-share based but though configuration of these controls a developer or administrator can limit or influence resource usage. Resources that can be controlled include: CPU/Processor, Memory/RAM, Disk/Storage and Networking/Throughput.

Windows containers utilize job objects to group and track processes associated with each container. Resource controls are implemented on the parent job object associated with the container.

In the case of Hyper-V isolation resource controls are applied both to the virtual machine as well as to the job object of the container running inside the virtual machine automatically, this ensures that even if a process running in the container bypassed or escaped the job objects controls the virtual machine would ensure it was not able to exceed the defined resource controls.

Resources

For each resource this section provides a mapping between the Docker command line interface as an example of how the resource control might be used (it might be configured by an orchestrator or other tooling) to the corresponding Windows host compute service (HCS) API as well as generally how the resource control has been implemented by Windows (please note that this description is high-level and that the underlying implementation is subject to change).

Memory

Resource Location
Docker interface --memory
HCS interface MemoryMaximumInMB
Shared Kernel JOB_OBJECT_LIMIT_JOB_MEMORY
Hyper-V isolation Virtual machine memory

Note

For Hyper-V isolation in Windows Server 2016, when using a memory cap, you'll see the container initially allocate the cap amount of memory, then start to return it to the container host. Later versions of Windows Server (1709 or later) have optimized this process.

CPU (count)

Resource Location
Docker interface --cpus
HCS interface ProcessorCount
Shared Kernel Simulated with JOB_OBJECT_CPU_RATE_CONTROL_HARD_CAP*
Hyper-V isolation Number of virtual processors exposed

CPU (percent)

Resource Location
Docker interface --cpu-percent
HCS interface ProcessorMaximum
Shared Kernel JOB_OBJECT_CPU_RATE_CONTROL_HARD_CAP
Hyper-V isolation Hypervisor limits on virtual processors

CPU (shares)

Resource Location
Docker interface --cpu-shares
HCS interface ProcessorWeight
Shared Kernel JOB_OBJECT_CPU_RATE_CONTROL_WEIGHT_BASED
Hyper-V isolation Hypervisor virtual processors weights

Storage (image)

Resource Location
Docker interface --io-maxbandwidth/--io-maxiops
HCS interface StorageIOPSMaximum and StorageBandwidthMaximum
Shared Kernel JOBOBJECT_IO_RATE_CONTROL_INFORMATION
Hyper-V isolation JOBOBJECT_IO_RATE_CONTROL_INFORMATION

Storage (volumes)

Resource Location
Docker interface --storage-opt size=
HCS interface StorageSandboxSize
Shared Kernel JOBOBJECT_IO_RATE_CONTROL_INFORMATION
Hyper-V isolation JOBOBJECT_IO_RATE_CONTROL_INFORMATION

Additional notes or details

Memory requirements

Windows containers run some system process in each container typically those which provide per-container functionality like user management, networking etc… and while much of the memory required by these processes is shared amongst containers the memory cap must be high enough to allow for them. A table is provided in the system requirements document for each base image type and with and without Hyper-V isolation.

CPU Shares (without Hyper-V isolation)

When using CPU shares the underlying implementation (when not using Hyper-V isolation) configures the JOBOBJECT_CPU_RATE_CONTROL_INFORMATION, specifically setting the control flag to JOB_OBJECT_CPU_RATE_CONTROL_WEIGHT_BASED and providing an appropriate Weight. The valid weight ranges of the job object are 1 – 9 with a default of 5 which is lower fidelity than the host compute services values of 1 – 10000. As examples a share weight of 7500 would result in a weight of 7 or a share weight of 2500 would result in a value of 2.