Windows container version compatibility
Windows Server 2016 and Windows 10 Anniversary Update (both version 14393) were the first Windows releases that could build and run Windows Server containers. Containers built using these versions can run on newer releases such as Windows Server version 1709, but there are a few things you need to know before you start.
As we've been improving the Windows container features, we've had to make some changes that can affect compatibility. Older containers will run the same on newer hosts with Hyper-V isolation, and will use the same (older) kernel version. However, if you want to run a container based on a newer Windows build, it can only run on the newer host build.
Note
* Windows Server, version 1709 is no longer supported. For more information, see Base image servicing lifecycles.
Windows Server, version 1903 host OS compatibility
| Container OS | Supports Hyper-V isolation | Supports process isolation |
|---|---|---|
| Windows Server, version 1903 | Yes | Yes |
| Windows Server 2019 | Yes | No |
| Windows Server, version 1803 | Yes | No |
| Windows Server, version 1709* | Yes | No |
| Windows Server 2016 | Yes | No |
Windows Server 2019 host OS compatibility
| Container OS | Supports Hyper-V isolation | Supports process isolation |
|---|---|---|
| Windows Server, version 1903 | No | No |
| Windows Server 2019 | Yes | Yes |
| Windows Server, version 1803 | Yes | No |
| Windows Server, version 1709* | Yes | No |
| Windows Server 2016 | Yes | No |
Windows Server, version 1803 host OS compatibility
| Container OS | Supports Hyper-V isolation | Supports process isolation |
|---|---|---|
| Windows Server, version 1903 | No | No |
| Windows Server 2019 | No | No |
| Windows Server, version 1803 | Yes | Yes |
| Windows Server, version 1709* | Yes | No |
| Windows Server 2016 | Yes | No |
Windows Server, version 1709 host OS compatibility*
| Container OS | Supports Hyper-V isolation | Supports process isolation |
|---|---|---|
| Windows Server, version 1903 | No | No |
| Windows Server 2019 | No | No |
| Windows Server, version 1803 | No | No |
| Windows Server, version 1709* | Yes | Yes |
| Windows Server 2016 | Yes | No |
Windows Server 2016 host OS compatibility
| Container OS | Supports Hyper-V isolation | Supports process isolation |
|---|---|---|
| Windows Server 2019, version 1903 | No | No |
| Windows Server 2019 | No | No |
| Windows Server, version 1803 | No | No |
| Windows Server, version 1709* | No | No |
| Windows Server 2016 | Yes | Yes |
Windows 10, version 1903 host OS compatibility
| Container OS | Supports Hyper-V isolation | Supports process isolation |
|---|---|---|
| Windows Server, version 1903 | Yes | No |
| Windows Server 2019 | Yes | No |
| Windows Server, version 1803 | Yes | No |
| Windows Server, version 1709* | Yes | No |
| Windows Server 2016 | Yes | No |
Windows 10, version 1809 host OS compatibility
| Container OS | Supports Hyper-V isolation | Supports process isolation |
|---|---|---|
| Windows Server, version 1903 | No | No |
| Windows Server 2019 | Yes | No |
| Windows Server, version 1803 | Yes | No |
| Windows Server, version 1709* | Yes | No |
| Windows Server 2016 | Yes | No |
Windows 10, version 1803 host OS compatibility
| Container OS | Supports Hyper-V isolation | Supports process isolation | |
|---|---|---|---|
| Windows Serve, version 1903 | No | No | |
| Windows Server 2019 | No | No | |
| Windows Server, version 1803 | Yes | No | |
| Windows Server, version 1709* | Yes | No | |
| Windows Server 2016 | Yes | No |
Windows 10 Fall Creators Update host OS compatibility
| Container OS | Supports Hyper-V isolation | Supports process isolation |
|---|---|---|
| Windows Server, version 1903 | No | No |
| Windows Server 2019 | No | No |
| Windows Server, version 1803 | No | No |
| Windows Server, version 1709* | Yes | No |
| Windows Server 2016 | Yes | No |
Matching container host version with container image versions
Windows Server containers
Because Windows Server containers and the underlying host share a single kernel, the container’s base image must match that of the host. If the versions are different, the container may start, but full functionally isn't guaranteed. The Windows operating system has four levels of versioning: major, minor, build and revision. For example, version 10.0.14393.103 would have a major version of 10, a minor version of 0, a build number of 14393, and a revision number of 103. The build number only changes when new versions of the OS are published, such as version 1709, 1803, Fall Creators' Update, and so on. The revision number is updated as Windows updates are applied.
Build number (new release of Windows)
Windows Server containers are blocked from starting when the build number between the container host and the container image are different. For example, when the container host is version 10.0.14393.* (Windows Server 2016) and container image is version 10.0.16299.* (Windows Server version 1709), the container won't start.
Revision number (patching)
Windows Server containers aren't blocked from starting when the revision numbers of the container host and the container image are different. For example, if the container host is version 10.0.14393.1914 (Windows Server 2016 with KB4051033 applied) and the container image is version 10.0.14393.1944 (Windows Server 2016 with KB4053579 applied), then the image will still start even though their revision numbers are different.
For Windows Server 2016-based hosts or images, the container image’s revision must match the host to be in a supported configuration. However, for hosts or images using Windows Server version 1709 and higher, this rule doesn't apply, and the host and container image need not have matching revisions. We recommend you keep your systems up-to-date with the latest patches and updates.
Practical application
Example 1: The container host is running Windows Server 2016 with KB4041691 applied. Any Windows Server container deployed to this host must be based on the version 10.0.14393.1770 container base images. If you apply KB4053579 to the host container, you must also update the images to make sure the host container supports them.
Example 2: The container host is running Windows Server version 1709 with KB4043961 applied. Any Windows Server container deployed to this host must be based on a Windows Server version 1709 (10.0.16299) container base image, but doesn't need to match the host KB. If KB4054517 is applied to the host, the container images will still be supported, but we recommend you update them to address any potential security issues.
Querying version
Method 1: Introduced in version 1709, the cmd prompt and ver command now return the revision details.
Microsoft Windows [Version 10.0.16299.125]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\>ver
Microsoft Windows [Version 10.0.16299.125]
Method 2: Query the following registry key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion
For example:
C:\>reg query "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion" /v BuildLabEx
Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
PS C:\Users\Administrator> (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\').BuildLabEx
14393.321.amd64fre.rs1_release_inmarket.161004-2338
To check what version your base image uses, review the tags on the Docker hub or the image hash table provided in the image description. The Windows 10 update history page lists when each build and revision was released.
Hyper-V isolation for containers
You can run Windows containers with or without Hyper-V isolation. Hyper-V isolation creates a secure boundary around the container with an optimized VM. Unlike standard Windows containers that share the kernel between containers and the host, each Hyper-V isolated container has its own instance of the Windows kernel. This means you can have different OS versions in the container host and image (for more information, see the following compatibility matrix).
To run a container with Hyper-V isolation, simply add the tag --isolation=hyperv to your docker run command.
Errors from mismatched versions
If you try to run an unsupported combination, you'll get the following error:
docker: Error response from daemon: container b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839 encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839","Owner":"docker","IsDummy":false,"VolumePath":"\\\\?\\Volume{2443d38a-1379-4bcf-a4b7-fc6ad4cd7b65}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839","Layers":[{"ID":"1532b584-8431-5b5a-8735-5e1b4fe9c2a9","Path":"C:\\ProgramData\\docker\\windowsfilter\\b2b88bc2a47abcc682e422507abbba9c9b6d826d34e67b9e4e3144cc125a1f80"},{"ID":"a64b8da5-cd6e-5540-bc73-d81acae6da54","Path":"C:\\ProgramData\\docker\\windowsfilter\\5caaedbced1f546bccd01c9d31ea6eea4d30701ebba7b95ee8faa8c098a6845a"}],"HostName":"b81ed896222e","MappedDirectories":[],"HvPartition":false,"EndpointList":["002a0d9e-13b7-42c0-89b2-c1e80d9af243"],"Servicing":false,"AllowUnqualifiedDNSQuery":true}.
There are three ways you can resolve this error:
- Rebuild the container based on the correct version of
mcr.microsoft.com/windows/nanoserverormcr.microsoft.com/windows/servercore - If the host is newer, run docker run --isolation=hyperv ...
- Try running the container on a different host with the same Windows version
Choose which container OS version to use
Note
As of April 16, 2019, the "latest" tag is no longer published or maintained for the Windows base OS container images. Please declare a specific tag when pulling or referencing images from these repos.
You must know which version you need to use for your container. For example, if you want Windows Server version 1809 as your container OS and want to have the latest patches for it, you should use the tag 1809 when specifying which version of the base OS container images you want, like so:
FROM mcr.microsoft.com/windows/nanoserver:1809
...
However, if you want a specific patch of Windows Server version 1809, you can specify the KB number in the tag. For example, to get a Nano Server base OS container image from Windows Server version 1809 with the KB4493509 applied to it, you would specify it like so:
FROM mcr.microsoft.com/windows/nanoserver:1809-KB4493509
...
You can also specify the exact patches you need with the schema we have used previously, by specifying the OS version in the tag:
FROM mcr.microsoft.com/windows/nanoserver:10.0.17763.437
...
The Server Core base images based on Windows Server 2019 and Windows Server 2016 are Long-Term Servicing Channel (LTSC) releases. If you for instance want Windows Server 2019 as your Server Core image's container OS and want to have the latest patches for it, you can specify LTSC releases like so:
FROM mcr.microsoft.com/windows/servercore:ltsc2019
...
Matching versions using Docker Swarm
Docker Swarm doesn't currently have a built-in way to match the version of Windows that a container uses to a host with the same version. If you update the service to use a newer container, it will run successfully.
If you need to run multiple versions of Windows for a long period of time, there are two approaches you can take: either configure the Windows hosts to always use Hyper-V isolation or use label constraints.
Finding a service that won't start
If a service won't start, you'll see that the MODE is replicated but REPLICAS will get stuck at 0. To see if the OS version is the problem, run the following commands:
Run docker service ls to find the service name:
ID NAME MODE REPLICAS IMAGE PORTS
xh6mwbdq2uil angry_liskov replicated 0/1 microsoft/iis:windowsservercore-10.0.14393.1715
Run docker service ps (service name) to get the status and latest attempts:
C:\Program Files\Docker>docker service ps angry_liskov
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
klkbhn742lv0 angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Ready Ready 3 seconds ago
y5blbdum70zo \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Shutdown Failed 24 seconds ago "starting container failed: co…"
yjq6zwzqj8kt \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Shutdown Failed 31 seconds ago "starting container failed: co…"
ytnnv80p03xx \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Shutdown Failed about a minute ago "starting container failed: co…"
xeqkxbsao57w \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Shutdown Failed about a minute ago "starting container failed: co…"
If you see starting container failed: ..., you can see the full error with docker service ps --no-trunc (container name):
C:\Program Files\Docker>docker service ps --no-trunc angry_liskov
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
dwsd6sjlwsgic5vrglhtxu178 angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715@sha256:868bca7e89e1743792e15f78edb5a73070ef44eae6807dc3f05f9b94c23943d5 WIN-BSTMQDRQC2E Running Starting less than a second ago
y5blbdum70zoh1f6uhx5nxsfv \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715@sha256:868bca7e89e1743792e15f78edb5a73070ef44eae6807dc3f05f9b94c23943d5 WIN-BSTMQDRQC2E Shutdown Failed 39 seconds ago "starting container failed: container e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0 encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0","Owner":"docker","VolumePath":"\\\\?\\Volume{2443d38a-1379-4bcf-a4b7-fc6ad4cd7b65}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0","Layers":[{"ID":"bcf2630f-ea95-529b-b33c-e5cdab0afdb4","Path":"C:\\ProgramData\\docker\\windowsfilter\\200235127f92416724ae1d53ed3fdc86d78767132d019bdda1e1192ee4cf3ae4"},{"ID":"e3ea10a8-4c2f-5b93-b2aa-720982f116f6","Path":"C:\\ProgramData\\docker\\windowsfilter\\0ccc9fa71a9f4c5f6f3bc8134fe3533e454e09f453de662cf99ab5d2106abbdc"},{"ID":"cff5391f-e481-593c-aff7-12e080c653ab","Path":"C:\\ProgramData\\docker\\windowsfilter\\a49576b24cd6ec4a26202871c36c0a2083d507394a3072186133131a72601a31"},{"ID":"499cb51e-b891-549a-b1f4-8a25a4665fbd","Path":"C:\\ProgramData\\docker\\windowsfilter\\fdf2f52c4323c62f7ff9b031c0bc3af42cf5fba91098d51089d039fb3e834c08"},{"ID":"1532b584-8431-5b5a-8735-5e1b4fe9c2a9","Path":"C:\\ProgramData\\docker\\windowsfilter\\b2b88bc2a47abcc682e422507abbba9c9b6d826d34e67b9e4e3144cc125a1f80"},{"ID":"a64b8da5-cd6e-5540-bc73-d81acae6da54","Path":"C:\\ProgramData\\docker\\windowsfilter\\5caaedbced1f546bccd01c9d31ea6eea4d30701ebba7b95ee8faa8c098a6845a"}],"HostName":"e7b5d3adba7e","HvPartition":false,"EndpointList":["298bb656-8800-4948-a41c-1b0500f3d94c"],"AllowUnqualifiedDNSQuery":true}"
This is the same error as CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101).
Fix - Update the service to use a matching version
There are two considerations for Docker Swarm. In the case where you have a compose file that has a service that uses an image you didn't create, you'll want to update the reference accordingly. For example:
version: '3'
services:
YourServiceName:
image: microsoft/windowsservercore:1709
...
The other consideration is if the image you are pointing to is one that you've created yourself (for example, contoso/myimage):
version: '3'
services:
YourServiceName:
image: contoso/myimage
...
In this case, you should use the method described in Errors from mismatched versions to modify that dockerfile instead of the docker-compose line.
Mitigation - Use Hyper-V isolation with Docker Swarm
There is a proposal to support using Hyper-V isolation on a per-container basis, but the code is not done yet. You can follow progress on GitHub. Until that's done, the hosts would need to be configured to always run with Hyper-V isolation.
This requires changing the Docker service configuration, then restarting the Docker engine.
Edit
C:\ProgramData\docker\config\daemon.jsonAdd a line with
"exec-opts":["isolation=hyperv"]Note
The daemon.json file does not exist by default. If you find that this is the case when you peek into the directory, you must create the file. Then you'll want to copy in the following:
{ "exec-opts":["isolation=hyperv"] }Close and save the file, then restart the docker engine by running the following cmdlets in PowerShell:
Stop-Service docker Start-Service dockerAfter you've restarted the service, launch your containers. Once they're running, you can verify the isolation level of a container by inspecting the container with the following cmdlet:
docker inspect --format='{{json .HostConfig.Isolation}}' $instanceNameOrId
It will return either "process" or "hyperv". If you have modified and set your daemon.json as described above, it should show the latter.
Mitigation - Use labels and constraints
Here's how to use labels and constraints to match versions:
Add labels to each node.
On each node, add two labels:
OSandOsVersion. This assumes you're running locally but could be modified to set them on a remote host instead.docker node update --label-add OS="windows" $ENV:COMPUTERNAME docker node update --label-add OsVersion="$((Get-ComputerInfo).OsVersion)" $ENV:COMPUTERNAMEAfterwards, you can check those by running the docker node inspect command, which should show the newly added labels:
"Spec": { "Labels": { "OS": "windows", "OsVersion": "10.0.16296" }, "Role": "manager", "Availability": "active" }Add a service constraint.
Now that you've labeled each node, you can update constraints that determine placement of services. In the following example, replace "contoso_service" with the name of your actual service:
docker service update \ --constraint-add "node.labels.OS == windows" \ --constraint-add "node.labels.OsVersion == $((Get-ComputerInfo).OsVersion)" \ contoso_serviceThis enforces and limits where a node may run.
To learn more about how to use service constraints, check out the service create reference.
Matching versions using Kubernetes
The same issue described in Matching versions using Docker Swarm can happen when pods are scheduled in Kubernetes. This issue can be avoided with similar strategies:
- Rebuild the container based on the same OS version in development and production. To learn how, see Choose which container OS version to use.
- Use node labels and nodeSelectors to make sure pods are scheduled on compatible nodes if both Windows Server 2016 and Windows Server version 1709 nodes are in the same cluster
- Use separate clusters based on OS version
Finding pods failed on OS mismatch
In this case, a deployment included a pod that was scheduled on a node with a mismatched OS version, and without Hyper-V isolation enabled.
The same error is shown in the events listed with kubectl describe pod <podname>. After several attempts, the pod status will probably be CrashLoopBackOff.
$ kubectl -n plang describe pod fabrikamfiber.web-789699744-rqv6p
Name: fabrikamfiber.web-789699744-rqv6p
Namespace: plang
Node: 38519acs9011/10.240.0.6
Start Time: Mon, 09 Oct 2017 19:40:30 +0000
Labels: io.kompose.service=fabrikamfiber.web
pod-template-hash=789699744
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"plang","name":"fabrikamfiber.web-789699744","uid":"b5062a08-ad29-11e7-b16e-000d3a...
Status: Running
IP: 10.244.3.169
Created By: ReplicaSet/fabrikamfiber.web-789699744
Controlled By: ReplicaSet/fabrikamfiber.web-789699744
Containers:
fabrikamfiberweb:
Container ID: docker://eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a
Image: patricklang/fabrikamfiber.web:latest
Image ID: docker-pullable://patricklang/fabrikamfiber.web@sha256:562741016ce7d9a232a389449a4fd0a0a55aab178cf324144404812887250ead
Port: 80/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: container eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a","Owner":"docker","IsDummy":false,"VolumePath":"\\\\?\\Volume{037b6606-bc9c-461f-ae02-829c28410798}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a","Layers":[{"ID":"f8bc427f-7aa3-59c6-b271-7331713e9451","Path":"C:\\ProgramData\\docker\\windowsfilter\\e206d2514a6265a76645b9d6b3dc6a78777c34dbf5da9fa2d564651645685881"},{"ID":"a6f35e41-a86c-5e4d-a19a-a6c2464bfb47","Path":"C:\\ProgramData\\docker\\windowsfilter\\0f21f1e28ef13030bbf0d87cbc97d1bc75f82ea53c842e9a3250a2156ced12d5"},{"ID":"4f624ca7-2c6d-5c42-b73f-be6e6baf2530","Path":"C:\\ProgramData\\docker\\windowsfilter\\4d9e2ad969fbd74fd58c98b5ab61e55a523087910da5200920e2b6f641d00c67"},{"ID":"88e360ff-32af-521d-9a3f-3760c12b35e2","Path":"C:\\ProgramData\\docker\\windowsfilter\\9e16a3d53d3e9b33344a6f0d4ed34c8a46448ee7636b672b61718225b8165e6e"},{"ID":"20f1a4e0-a6f3-5db3-9bf2-01fd3e9e855a","Path":"C:\\ProgramData\\docker\\windowsfilter\\7eec7f59f9adce38cc0a6755da58a3589287d920d37414b5b21b5b543d910461"},{"ID":"c2b3d728-4879-5343-a92a-b735752a4724","Path":"C:\\ProgramData\\docker\\windowsfilter\\8ed04b60acc0f65f541292a9e598d5f73019c8db425f8d49ea5819eab16a42f3"},{"ID":"2973e760-dc59-5800-a3de-ab9d93be81e5","Path":"C:\\ProgramData\\docker\\windowsfilter\\cc71305d45f09ce377ef497f28c3a74ee027c27f20657d2c4a5f157d2457cc75"},{"ID":"454a7d36-038c-5364-8a25-fa84091869d6","Path":"C:\\ProgramData\\docker\\windowsfilter\\9e8cde1ce8f5de861a5f22841f1ab9bc53d5f606d06efeacf5177f340e8d54d0"},{"ID":"9b748c8c-69eb-55fb-a1c1-5688cac4efd8","Path":"C:\\ProgramData\\docker\\windowsfilter\\8e02bf5404057055a71d542780a2bb2731be4b3707c01918ba969fb4d83b98ec"},{"ID":"bfde5c26-b33f-5424-9405-9d69c2fea4d0","Path":"C:\\ProgramData\\docker\\windowsfilter\\77483cedfb0964008c33d92d306734e1fab3b5e1ebb27e898f58ccfd108108f2"},{"ID":"bdabfbf5-80d1-57f1-86f3-448ce97e2d05","Path":"C:\\ProgramData\\docker\\windowsfilter\\aed2ebbb31ad24b38ee8521dd17744319f83d267bf7f360bc177e27ae9a006cf"},{"ID":"ad9b34f2-dcee-59ea-8962-b30704ae6331","Path":"C:\\ProgramData\\docker\\windowsfilter\\d44d3a675fec1070b61d6ea9bacef4ac12513caf16bd6453f043eed2792f75d8"}],"HostName":"fabrikamfiber.web-789699744-rqv6p","MappedDirectories":[{"HostPath":"c:\\var\\lib\\kubelet\\pods\\b50f0027-ad29-11e7-b16e-000d3afd2878\\volumes\\kubernetes.io~secret\\default-token-rw9dn","ContainerPath":"c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0}],"HvPartition":false,"EndpointList":null,"NetworkSharedContainerName":"586870f5833279678773cb700db3c175afc81d557a75867bf39b43f985133d13","Servicing":false,"AllowUnqualifiedDNSQuery":false}
Exit Code: 128
Started: Mon, 09 Oct 2017 20:27:08 +0000
Finished: Mon, 09 Oct 2017 20:27:08 +0000
Ready: False
Restart Count: 10
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rw9dn (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-rw9dn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rw9dn
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=windows
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
49m 49m 1 default-scheduler Normal Scheduled Successfully assigned fabrikamfiber.web-789699744-rqv6p to 38519acs9011
49m 49m 1 kubelet, 38519acs9011 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-rw9dn"
29m 29m 1 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning Failed Failed to pull image "patricklang/fabrikamfiber.web:latest": rpc error: code = 2 desc = Error response from daemon: {"message":"Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: no such host"}
49m 3m 12 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Pulling pulling image "patricklang/fabrikamfiber.web:latest"
33m 3m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Pulled Successfully pulled image "patricklang/fabrikamfiber.web:latest"
33m 3m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Created Created container
33m 2m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning Failed Error: failed to start container "fabrikamfiberweb": Error response from daemon: {"message":"container fabrikamfiberweb encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {\"SystemType\":\"Container\",\"Name\":\"fabrikamfiberweb\",\"Owner\":\"docker\",\"IsDummy\":false,\"VolumePath\":\"\\\\\\\\?\\\\Volume{037b6606-bc9c-461f-ae02-829c28410798}\",\"IgnoreFlushesDuringBoot\":true,\"LayerFolderPath\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\fabrikamfiberweb\",\"Layers\":[{\"ID\":\"f8bc427f-7aa3-59c6-b271-7331713e9451\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\e206d2514a6265a76645b9d6b3dc6a78777c34dbf5da9fa2d564651645685881\"},{\"ID\":\"a6f35e41-a86c-5e4d-a19a-a6c2464bfb47\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\0f21f1e28ef13030bbf0d87cbc97d1bc75f82ea53c842e9a3250a2156ced12d5\"},{\"ID\":\"4f624ca7-2c6d-5c42-b73f-be6e6baf2530\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\4d9e2ad969fbd74fd58c98b5ab61e55a523087910da5200920e2b6f641d00c67\"},{\"ID\":\"88e360ff-32af-521d-9a3f-3760c12b35e2\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\9e16a3d53d3e9b33344a6f0d4ed34c8a46448ee7636b672b61718225b8165e6e\"},{\"ID\":\"20f1a4e0-a6f3-5db3-9bf2-01fd3e9e855a\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\7eec7f59f9adce38cc0a6755da58a3589287d920d37414b5b21b5b543d910461\"},{\"ID\":\"c2b3d728-4879-5343-a92a-b735752a4724\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\8ed04b60acc0f65f541292a9e598d5f73019c8db425f8d49ea5819eab16a42f3\"},{\"ID\":\"2973e760-dc59-5800-a3de-ab9d93be81e5\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\cc71305d45f09ce377ef497f28c3a74ee027c27f20657d2c4a5f157d2457cc75\"},{\"ID\":\"454a7d36-038c-5364-8a25-fa84091869d6\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\9e8cde1ce8f5de861a5f22841f1ab9bc53d5f606d06efeacf5177f340e8d54d0\"},{\"ID\":\"9b748c8c-69eb-55fb-a1c1-5688cac4efd8\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\8e02bf5404057055a71d542780a2bb2731be4b3707c01918ba969fb4d83b98ec\"},{\"ID\":\"bfde5c26-b33f-5424-9405-9d69c2fea4d0\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\77483cedfb0964008c33d92d306734e1fab3b5e1ebb27e898f58ccfd108108f2\"},{\"ID\":\"bdabfbf5-80d1-57f1-86f3-448ce97e2d05\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\aed2ebbb31ad24b38ee8521dd17744319f83d267bf7f360bc177e27ae9a006cf\"},{\"ID\":\"ad9b34f2-dcee-59ea-8962-b30704ae6331\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\d44d3a675fec1070b61d6ea9bacef4ac12513caf16bd6453f043eed2792f75d8\"}],\"HostName\":\"fabrikamfiber.web-789699744-rqv6p\",\"MappedDirectories\":[{\"HostPath\":\"c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\b50f0027-ad29-11e7-b16e-000d3afd2878\\\\volumes\\\\kubernetes.io~secret\\\\default-token-rw9dn\",\"ContainerPath\":\"c:\\\\var\\\\run\\\\secrets\\\\kubernetes.io\\\\serviceaccount\",\"ReadOnly\":true,\"BandwidthMaximum\":0,\"IOPSMaximum\":0}],\"HvPartition\":false,\"EndpointList\":null,\"NetworkSharedContainerName\":\"586870f5833279678773cb700db3c175afc81d557a75867bf39b43f985133d13\",\"Servicing\":false,\"AllowUnqualifiedDNSQuery\":false}"}
33m 11s 151 kubelet, 38519acs9011 Warning FailedSync Error syncing pod
32m 11s 139 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning BackOff Back-off restarting failed container
Mitigation - using node labels and nodeSelector
Run kubectl get node to get a list of all nodes. After that, you can run kubectl describe node (node name) to get more details.
In the following example, two Windows nodes are running different versions:
$ kubectl get node
NAME STATUS AGE VERSION
38519acs9010 Ready 21h v1.7.7-7+e79c96c8ff2d8e
38519acs9011 Ready 4h v1.7.7-25+bc3094f1d650a2
k8s-linuxpool1-38519084-0 Ready 21h v1.7.7
k8s-master-38519084-0 Ready 21h v1.7.7
$ kubectl describe node 38519acs9010
Name: 38519acs9010
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_D2_v2
beta.kubernetes.io/os=windows
failure-domain.beta.kubernetes.io/region=westus2
failure-domain.beta.kubernetes.io/zone=0
kubernetes.io/hostname=38519acs9010
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Fri, 06 Oct 2017 01:41:02 +0000
...
System Info:
Machine ID: 38519acs9010
System UUID:
Boot ID:
Kernel Version: 10.0 14393 (14393.1715.amd64fre.rs1_release_inmarket.170906-1810)
OS Image:
Operating System: windows
Architecture: amd64
...
$ kubectl describe node 38519acs9011
Name: 38519acs9011
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_DS1_v2
beta.kubernetes.io/os=windows
failure-domain.beta.kubernetes.io/region=westus2
failure-domain.beta.kubernetes.io/zone=0
kubernetes.io/hostname=38519acs9011
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Fri, 06 Oct 2017 18:13:25 +0000
Conditions:
...
System Info:
Machine ID: 38519acs9011
System UUID:
Boot ID:
Kernel Version: 10.0 16299 (16299.0.amd64fre.rs3_release.170922-1354)
OS Image:
Operating System: windows
Architecture: amd64
...
Let's use this example to show how to match the versions:
Take note of each node name and
Kernel Versionfrom the system info.In our example, the info will look like this:
Name Version 38519acs9010 14393.1715.amd64fre.rs1_release_inmarket.170906-1810 38519acs9011 16299.0.amd64fre.rs3_release.170922-1354 Add a label to each node called
beta.kubernetes.io/osbuild. Windows Server 2016 needs both major and minor versions (14393.1715 in this example) to be supported without Hyper-V isolation. Windows Server version 1709 only needs the major version (16299 in this example) to match.In this example, the command to add the labels looks like this:
$ kubectl label node 38519acs9010 beta.kubernetes.io/osbuild=14393.1715 node "38519acs9010" labeled $ kubectl label node 38519acs9011 beta.kubernetes.io/osbuild=16299 node "38519acs9011" labeledCheck the labels are there by running kubectl get nodes --show-labels.
In this example, the output will look like this:
$ kubectl get nodes --show-labels NAME STATUS AGE VERSION LABELS 38519acs9010 Ready,SchedulingDisabled 3d v1.7.7-7+e79c96c8ff2d8e beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=windows,beta.kubernetes.io/osbuild=14393.1715,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=38519acs9010 38519acs9011 Ready 3d v1.7.7-25+bc3094f1d650a2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_DS1_v2,beta.kubernetes.io/os=windows,beta.kubernetes.io/osbuild=16299,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=38519acs9011 k8s-linuxpool1-38519084-0 Ready 3d v1.7.7 agentpool=linuxpool1,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=k8s-linuxpool1-38519084-0,kubernetes.io/role=agent k8s-master-38519084-0 Ready 3d v1.7.7 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=k8s-master-38519084-0,kubernetes.io/role=masterAdd node selectors to deployments. In this example case, we'll add a
nodeSelectorto the container spec withbeta.kubernetes.io/os= windows andbeta.kubernetes.io/osbuild= 14393.* or 16299 to match the base OS used by the container.Here's a full example for running a container built for Windows Server 2016:
apiVersion: extensions/v1beta1 kind: Deployment metadata: annotations: kompose.cmd: kompose convert -f docker-compose-combined.yml kompose.version: 1.2.0 (99f88ef) creationTimestamp: null labels: io.kompose.service: fabrikamfiber.web name: fabrikamfiber.web spec: replicas: 1 strategy: {} template: metadata: creationTimestamp: null labels: io.kompose.service: fabrikamfiber.web spec: containers: - image: patricklang/fabrikamfiber.web:latest name: fabrikamfiberweb ports: - containerPort: 80 resources: {} restartPolicy: Always nodeSelector: "beta.kubernetes.io/os": windows "beta.kubernetes.io/osbuild": "14393.1715" status: {}The pod can now start with the updated deployment. The node selectors are also shown in
kubectl describe pod <podname>, so you can run that command to verify they were added.The output for our example is as follows:
$ kubectl -n plang describe po fa Name: fabrikamfiber.web-1780117715-5c8vw Namespace: plang Node: 38519acs9010/10.240.0.4 Start Time: Tue, 10 Oct 2017 01:43:28 +0000 Labels: io.kompose.service=fabrikamfiber.web pod-template-hash=1780117715 Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"plang","name":"fabrikamfiber.web-1780117715","uid":"6a07aaf3-ad5c-11e7-b16e-000d3... Status: Running IP: 10.244.1.84 Created By: ReplicaSet/fabrikamfiber.web-1780117715 Controlled By: ReplicaSet/fabrikamfiber.web-1780117715 Containers: fabrikamfiberweb: Container ID: docker://c94594fb53161f3821cf050d9af7546991aaafbeab41d333d9f64291327fae13 Image: patricklang/fabrikamfiber.web:latest Image ID: docker-pullable://patricklang/fabrikamfiber.web@sha256:562741016ce7d9a232a389449a4fd0a0a55aab178cf324144404812887250ead Port: 80/TCP State: Running Started: Tue, 10 Oct 2017 01:43:42 +0000 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-rw9dn (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: default-token-rw9dn: Type: Secret (a volume populated by a Secret) SecretName: default-token-rw9dn Optional: false QoS Class: BestEffort Node-Selectors: beta.kubernetes.io/os=windows beta.kubernetes.io/osbuild=14393.1715 Tolerations: <none> Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 5m 5m 1 default-scheduler Normal Scheduled Successfully assigned fabrikamfiber.web-1780117715-5c8vw to 38519acs9010 5m 5m 1 kubelet, 38519acs9010 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-rw9dn" 5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Pulling pulling image "patricklang/fabrikamfiber.web:latest" 5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Pulled Successfully pulled image "patricklang/fabrikamfiber.web:latest" 5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Created Created container 5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Started Started container
Feedback
Loading feedback...