Windows Container Version Compatibility
- 14 minutes to read
-
Contributors
Windows Server 2016 and Windows 10 Anniversary Update (both version 14393) were the first Windows releases that could build and run Windows Server Containers. Containers built using these versions can run on newer releases such as Windows Server version 1709, but there are a few things you need to know before you start.
As we've been improving the Windows container features, we've had to make some changes that can affect compatibility. Older containers will run the same on newer hosts with Hyper-V isolation, and the same (older) kernel version will still be used. However, if you want to run a container based on a newer Windows build - it can only run on the newer host build.
| Container OS version | Host OS version | |||
|---|---|---|---|---|
| Windows Server 2016 Builds: 14393. |
Windows 10 1609, 1703 Builds: 14393., 15063. |
Windows Server version 1709 Builds 16299. |
Windows 10 Fall Creators Update Builds 16299. |
|
| Windows Server 2016 Builds: 14393. |
Supportsprocess or hyperv isolation |
Supports Only hyperv isolation |
Supports Only hyperv isolation |
Supports Only hyperv isolation |
| Windows Server version 1709 Builds 16299.* |
Not supported | Not supported | Supportsprocess or hyperv isolation |
Supports Only hyperv isolation |
Matching Container Host Version with Container Image Versions
Windows Server Containers
Because Windows Server Containers and the underlying host share a single kernel, the container’s base image must match that of the host. If the versions are different the container may start, but full functionally cannot be guaranteed. The Windows operating system has 4 levels of versioning, Major, Minor, Build and Revision – for example 10.0.14393.103. The build number (i.e. 14393) only changes when new versions of the OS are published, such as version 1709, 1803, fall creators update etc... The revision number (i.e. 103) is updated as Windows updates are applied.
Build Number (new release of Windows)
Windows Server Containers are blocked from starting when the build number between the container host and the container image are different - for example 10.0.14393.* (Windows Server 2016) and 10.0.16299.* (Windows Server version 1709).
Revision Number (patching)
Windows Server Containers are not blocked from starting when the revision number between the container host and the container image are different for example 10.0.14393.1914 (Windows Server 2016 with KB4051033 applied) and 10.0.14393.1944 (Windows Server 2016 with KB4053579 applied).
For Windows Server 2016 based hosts/images – the container image’s revision must match the host to be in a supported configuration. Starting with Windows Server version 1709, this is no longer applicable, and the host and container image need not have matching revisions. It is as always recommended to keep your systems up-to-date with the latest patches and updates.
Practical Application
Example 1: Container host is running Windows Server 2016 with KB4041691 applied. Any Windows Server container deployed to this host must be based on the 10.0.14393.1770 container base images. If KB4053579 is applied to the host the container images must be updated at the same time to remain supported. Example 2: Container host is running Windows Server version 1709 with KB4043961 applied. Any Windows Server container deployed to this host must be based on a Windows Server version 1709 (10.0.16299) container base image but need not match the host KB. If KB4054517 is applied to the host the container images do not need to be updated, though should be in order to fully address any security issues.
Querying version
Method 1:
Introduced in version 1709 the cmd prompt and ver command now return the revision details.
Microsoft Windows [Version 10.0.16299.125]
(c) 2017 Microsoft Corporation. All rights reserved.
C:\>ver
Microsoft Windows [Version 10.0.16299.125]
Method 2: Query the following registry key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion For example:
C:\>reg query "HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion" /v BuildLabEx
Or
Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.
PS C:\Users\Administrator> (Get-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\').BuildLabEx
14393.321.amd64fre.rs1_release_inmarket.161004-2338
To check what version your base image is using you can review the tags on the Docker hub or the image hash table provided in the image description. The Windows 10 Update History page lists when each build and revision was released.
Hyper-V Isolation for Containers
Windows containers can be run with or without Hyper-V isolation. Hyper-V isolation creates a secure boundary around the container with an optimized VM. Unlike standard Windows containers, which share the kernel between containers and the host, each Hyper-V isolated container has its own instance of the Windows kernel. Because of this you can have different OS versions in the container host and image (see compatibility matrix below).
To run a container with Hyper-V isolation, simply add the tag --isolation=hyperv to your docker run command.
Errors from mismatched versions
If you try to run an unsupported combination - you'll get an error:
docker: Error response from daemon: container b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839 encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839","Owner":"docker","IsDummy":false,"VolumePath":"\\\\?\\Volume{2443d38a-1379-4bcf-a4b7-fc6ad4cd7b65}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\b81ed896222eb87906ccab1c3dd2fc49324eafa798438f7979b87b210906f839","Layers":[{"ID":"1532b584-8431-5b5a-8735-5e1b4fe9c2a9","Path":"C:\\ProgramData\\docker\\windowsfilter\\b2b88bc2a47abcc682e422507abbba9c9b6d826d34e67b9e4e3144cc125a1f80"},{"ID":"a64b8da5-cd6e-5540-bc73-d81acae6da54","Path":"C:\\ProgramData\\docker\\windowsfilter\\5caaedbced1f546bccd01c9d31ea6eea4d30701ebba7b95ee8faa8c098a6845a"}],"HostName":"b81ed896222e","MappedDirectories":[],"HvPartition":false,"EndpointList":["002a0d9e-13b7-42c0-89b2-c1e80d9af243"],"Servicing":false,"AllowUnqualifiedDNSQuery":true}.
To resolve this, you could:
- Rebuild the container based on the version of
microsoft/nanoserverormicrosoft/windowsservercore - If the host is newer, use
docker run --isolation=hyperv ... - Run on a different host with the same Windows version
Choosing Container OS versions
Note: The "latest" tag will be updated along with Windows Server 2016, the current LTSC product. If you would like the container images that match the Windows Server version 1709 release, read below.
It is important to ensure you know what Container OS version you will need for your purposes. If you are using Windows Server version 1709, and want to have the latest patches for it, you should use the tag "1709" when specifying which version of the base OS container images you want, like so:
FROM microsoft/windowsservercore:1709
...
However, if you would like a specific patch of Windows Server version 1709, you can specify the KB number in the tag. For example, if you would like the Nano Server base OS container image from Windows Server version 1709, with the KB4043961 applied to it, you would specified it like so:
FROM microsoft/nanoserver:1709_KB4043961
...
And if you need the Nano Server base OS container image from Windows Server 2016, you can still get the latest version of those base OS container images by using the "latest" tag:
FROM microsoft/nanoserver
...
You can also continue specifying the exact patches you need with the schema we have used previously, by specifying the OS version in the tag:
FROM microsoft/nanoserver:10.0.14393.1770
...
Matching versions using Docker Swarm
Today, Docker Swarm does not have a built-in way to match the version of Windows that a container uses to a host matching the same version. If the service is updated to use a newer container, it will run successfully.
If you need to run multiple versions of Windows for some time, then there are two approaches that can be used. Configure the Windows hosts to always use Hyper-V isolation, or use label constraints.
Finding a service that won't start
If a service won't start - you'll see that the MODE is replicated but REPLICAS will get stuck at 0. To see if the OS version is the problem, then use the following commands:
docker service ls - find the service name
ID NAME MODE REPLICAS IMAGE PORTS
xh6mwbdq2uil angry_liskov replicated 0/1 microsoft/iis:windowsservercore-10.0.14393.1715
docker service ps <name> - get the status and latest attempts.
C:\Program Files\Docker>docker service ps angry_liskov
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
klkbhn742lv0 angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Ready Ready 3 seconds ago
y5blbdum70zo \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Shutdown Failed 24 seconds ago "starting container failed: co…"
yjq6zwzqj8kt \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Shutdown Failed 31 seconds ago "starting container failed: co…"
ytnnv80p03xx \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Shutdown Failed about a minute ago "starting container failed: co…"
xeqkxbsao57w \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715 WIN-BSTMQDRQC2E Shutdown Failed about a minute ago "starting container failed: co…"
If you see "starting container failed: ...", you can see the full error with docker service ps --no-trunc <container name>
C:\Program Files\Docker>docker service ps --no-trunc angry_liskov
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
dwsd6sjlwsgic5vrglhtxu178 angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715@sha256:868bca7e89e1743792e15f78edb5a73070ef44eae6807dc3f05f9b94c23943d5 WIN-BSTMQDRQC2E Running Starting less than a second ago
y5blbdum70zoh1f6uhx5nxsfv \_ angry_liskov.1 microsoft/iis:windowsservercore-10.0.14393.1715@sha256:868bca7e89e1743792e15f78edb5a73070ef44eae6807dc3f05f9b94c23943d5 WIN-BSTMQDRQC2E Shutdown Failed 39 seconds ago "starting container failed: container e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0 encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0","Owner":"docker","VolumePath":"\\\\?\\Volume{2443d38a-1379-4bcf-a4b7-fc6ad4cd7b65}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\e7b5d3adba7e510569c18d8e55f7c689d7cb92be40a516c91b363e27f84604d0","Layers":[{"ID":"bcf2630f-ea95-529b-b33c-e5cdab0afdb4","Path":"C:\\ProgramData\\docker\\windowsfilter\\200235127f92416724ae1d53ed3fdc86d78767132d019bdda1e1192ee4cf3ae4"},{"ID":"e3ea10a8-4c2f-5b93-b2aa-720982f116f6","Path":"C:\\ProgramData\\docker\\windowsfilter\\0ccc9fa71a9f4c5f6f3bc8134fe3533e454e09f453de662cf99ab5d2106abbdc"},{"ID":"cff5391f-e481-593c-aff7-12e080c653ab","Path":"C:\\ProgramData\\docker\\windowsfilter\\a49576b24cd6ec4a26202871c36c0a2083d507394a3072186133131a72601a31"},{"ID":"499cb51e-b891-549a-b1f4-8a25a4665fbd","Path":"C:\\ProgramData\\docker\\windowsfilter\\fdf2f52c4323c62f7ff9b031c0bc3af42cf5fba91098d51089d039fb3e834c08"},{"ID":"1532b584-8431-5b5a-8735-5e1b4fe9c2a9","Path":"C:\\ProgramData\\docker\\windowsfilter\\b2b88bc2a47abcc682e422507abbba9c9b6d826d34e67b9e4e3144cc125a1f80"},{"ID":"a64b8da5-cd6e-5540-bc73-d81acae6da54","Path":"C:\\ProgramData\\docker\\windowsfilter\\5caaedbced1f546bccd01c9d31ea6eea4d30701ebba7b95ee8faa8c098a6845a"}],"HostName":"e7b5d3adba7e","HvPartition":false,"EndpointList":["298bb656-8800-4948-a41c-1b0500f3d94c"],"AllowUnqualifiedDNSQuery":true}"
Which shows the same error CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101)
Fix - Update the service to use a matching version
There are two considerations for Docker Swarm. In the case where you have a compose file which has a service that uses an image which is not created by you, you'll want to update the reference accordingly. See below:
version: '3'
services:
YourServiceName:
image: microsoft/windowsservercore:1709
...
The other consideration is if the image you are pointing to is one that you've created yourself (say, contoso/myimage):
version: '3'
services:
YourServiceName:
image: contoso/myimage
...
In this case, you would want to use the method described in the previous section to modify that dockerfile instead of the docker-compose line.
Mitigation - Use Hyper-V isolation with Docker Swarm
There is a proposal to support using Hyper-V isolation on a per container basis, but the code is not done yet. You can follow progress on GitHub. Until that's done, the hosts would need to be configured to always run with Hyper-V isolation.
This requires changing the Docker service configuration, then restarting the Docker engine.
- Edit
C:\ProgramData\docker\config\daemon.json - Add a line with
"exec-opts":["isolation=hyperv"]
NOTE: The daemon.json file does not exist by default. If you find that this is the case when you peek into the directory, you must create the file. Then you'll want to copy in the content below:
{
"exec-opts":["isolation=hyperv"]
}
Close and save the file. Then restart the docker engine:
Stop-Service docker
Start-Service docker
After the service is restarted, launch your containers. Once they're running, you can verify the isolation level of a container by inspecting the container:
docker inspect --format='{{json .HostConfig.Isolation}}' $instanceNameOrId
It will return either "process" or "hyperv". If you have modified and set your daemon.json as described above, it should show the latter.
Mitigation - Use labels and constraints
Step 1 - Add labels to each node
On each node, add two labels - OS and OsVersion. This assumes you're running locally but could be modified to set them on a remote host instead.
docker node update --label-add OS="windows" $ENV:COMPUTERNAME
docker node update --label-add OsVersion="$((Get-ComputerInfo).OsVersion)" $ENV:COMPUTERNAME
Afterwards, you can check those with docker node inspect which will show the new labels added
"Spec": {
"Labels": {
"OS": "windows",
"OsVersion": "10.0.16296"
},
"Role": "manager",
"Availability": "active"
}
Step 2 - Add a service constraint
Once each node has been labeled, we can update constraints which determine placement of services. In the example below, substitude "contoso_service" with the name of your actual service:
docker service update \
--constraint-add "node.labels.OS == windows" \
--constraint-add "node.labels.OsVersion == $((Get-ComputerInfo).OsVersion)" \
contoso_service
This enforces and limits where a node may run.
For more details on how to use service constraints, check out the service create reference
Matching versions using Kubernetes
The same issue can happen when pods are scheduled in Kubernetes. This can be avoided using similar strategies:
- Rebuild the container based on the same OS version in development and production - see Choosing Container OS versions above
- Use node labels & nodeSelectors to make sure pods are scheduled on compatible nodes if both Windows Server 2016 and Windows Server version 1709 nodes are in the same cluster
- Use separate clusters based on OS version
Finding pods failed on OS mismatch
In this case - a deployment included a pod that was scheduled on a node with a mismatched OS version, and without Hyper-V isolation enabled.
The same error is shown in the events listed with kubectl describe pod <podname>. After several attempts, the pod status will probably be CrashLoopBackOff
$ kubectl -n plang describe po fabrikamfiber.web-789699744-rqv6p
Name: fabrikamfiber.web-789699744-rqv6p
Namespace: plang
Node: 38519acs9011/10.240.0.6
Start Time: Mon, 09 Oct 2017 19:40:30 +0000
Labels: io.kompose.service=fabrikamfiber.web
pod-template-hash=789699744
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"plang","name":"fabrikamfiber.web-789699744","uid":"b5062a08-ad29-11e7-b16e-000d3a...
Status: Running
IP: 10.244.3.169
Created By: ReplicaSet/fabrikamfiber.web-789699744
Controlled By: ReplicaSet/fabrikamfiber.web-789699744
Containers:
fabrikamfiberweb:
Container ID: docker://eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a
Image: patricklang/fabrikamfiber.web:latest
Image ID: docker-pullable://patricklang/fabrikamfiber.web@sha256:562741016ce7d9a232a389449a4fd0a0a55aab178cf324144404812887250ead
Port: 80/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: ContainerCannotRun
Message: container eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {"SystemType":"Container","Name":"eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a","Owner":"docker","IsDummy":false,"VolumePath":"\\\\?\\Volume{037b6606-bc9c-461f-ae02-829c28410798}","IgnoreFlushesDuringBoot":true,"LayerFolderPath":"C:\\ProgramData\\docker\\windowsfilter\\eab0151378308315ed6c31adf4ad9e31e6d65fd300e56e742757004a969a803a","Layers":[{"ID":"f8bc427f-7aa3-59c6-b271-7331713e9451","Path":"C:\\ProgramData\\docker\\windowsfilter\\e206d2514a6265a76645b9d6b3dc6a78777c34dbf5da9fa2d564651645685881"},{"ID":"a6f35e41-a86c-5e4d-a19a-a6c2464bfb47","Path":"C:\\ProgramData\\docker\\windowsfilter\\0f21f1e28ef13030bbf0d87cbc97d1bc75f82ea53c842e9a3250a2156ced12d5"},{"ID":"4f624ca7-2c6d-5c42-b73f-be6e6baf2530","Path":"C:\\ProgramData\\docker\\windowsfilter\\4d9e2ad969fbd74fd58c98b5ab61e55a523087910da5200920e2b6f641d00c67"},{"ID":"88e360ff-32af-521d-9a3f-3760c12b35e2","Path":"C:\\ProgramData\\docker\\windowsfilter\\9e16a3d53d3e9b33344a6f0d4ed34c8a46448ee7636b672b61718225b8165e6e"},{"ID":"20f1a4e0-a6f3-5db3-9bf2-01fd3e9e855a","Path":"C:\\ProgramData\\docker\\windowsfilter\\7eec7f59f9adce38cc0a6755da58a3589287d920d37414b5b21b5b543d910461"},{"ID":"c2b3d728-4879-5343-a92a-b735752a4724","Path":"C:\\ProgramData\\docker\\windowsfilter\\8ed04b60acc0f65f541292a9e598d5f73019c8db425f8d49ea5819eab16a42f3"},{"ID":"2973e760-dc59-5800-a3de-ab9d93be81e5","Path":"C:\\ProgramData\\docker\\windowsfilter\\cc71305d45f09ce377ef497f28c3a74ee027c27f20657d2c4a5f157d2457cc75"},{"ID":"454a7d36-038c-5364-8a25-fa84091869d6","Path":"C:\\ProgramData\\docker\\windowsfilter\\9e8cde1ce8f5de861a5f22841f1ab9bc53d5f606d06efeacf5177f340e8d54d0"},{"ID":"9b748c8c-69eb-55fb-a1c1-5688cac4efd8","Path":"C:\\ProgramData\\docker\\windowsfilter\\8e02bf5404057055a71d542780a2bb2731be4b3707c01918ba969fb4d83b98ec"},{"ID":"bfde5c26-b33f-5424-9405-9d69c2fea4d0","Path":"C:\\ProgramData\\docker\\windowsfilter\\77483cedfb0964008c33d92d306734e1fab3b5e1ebb27e898f58ccfd108108f2"},{"ID":"bdabfbf5-80d1-57f1-86f3-448ce97e2d05","Path":"C:\\ProgramData\\docker\\windowsfilter\\aed2ebbb31ad24b38ee8521dd17744319f83d267bf7f360bc177e27ae9a006cf"},{"ID":"ad9b34f2-dcee-59ea-8962-b30704ae6331","Path":"C:\\ProgramData\\docker\\windowsfilter\\d44d3a675fec1070b61d6ea9bacef4ac12513caf16bd6453f043eed2792f75d8"}],"HostName":"fabrikamfiber.web-789699744-rqv6p","MappedDirectories":[{"HostPath":"c:\\var\\lib\\kubelet\\pods\\b50f0027-ad29-11e7-b16e-000d3afd2878\\volumes\\kubernetes.io~secret\\default-token-rw9dn","ContainerPath":"c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount","ReadOnly":true,"BandwidthMaximum":0,"IOPSMaximum":0}],"HvPartition":false,"EndpointList":null,"NetworkSharedContainerName":"586870f5833279678773cb700db3c175afc81d557a75867bf39b43f985133d13","Servicing":false,"AllowUnqualifiedDNSQuery":false}
Exit Code: 128
Started: Mon, 09 Oct 2017 20:27:08 +0000
Finished: Mon, 09 Oct 2017 20:27:08 +0000
Ready: False
Restart Count: 10
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rw9dn (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
default-token-rw9dn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rw9dn
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=windows
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
49m 49m 1 default-scheduler Normal Scheduled Successfully assigned fabrikamfiber.web-789699744-rqv6p to 38519acs9011
49m 49m 1 kubelet, 38519acs9011 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-rw9dn"
29m 29m 1 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning Failed Failed to pull image "patricklang/fabrikamfiber.web:latest": rpc error: code = 2 desc = Error response from daemon: {"message":"Get https://registry-1.docker.io/v2/: dial tcp: lookup registry-1.docker.io: no such host"}
49m 3m 12 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Pulling pulling image "patricklang/fabrikamfiber.web:latest"
33m 3m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Pulled Successfully pulled image "patricklang/fabrikamfiber.web:latest"
33m 3m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Normal Created Created container
33m 2m 11 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning Failed Error: failed to start container "fabrikamfiberweb": Error response from daemon: {"message":"container fabrikamfiberweb encountered an error during CreateContainer: failure in a Windows system call: The operating system of the container does not match the operating system of the host. (0xc0370101) extra info: {\"SystemType\":\"Container\",\"Name\":\"fabrikamfiberweb\",\"Owner\":\"docker\",\"IsDummy\":false,\"VolumePath\":\"\\\\\\\\?\\\\Volume{037b6606-bc9c-461f-ae02-829c28410798}\",\"IgnoreFlushesDuringBoot\":true,\"LayerFolderPath\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\fabrikamfiberweb\",\"Layers\":[{\"ID\":\"f8bc427f-7aa3-59c6-b271-7331713e9451\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\e206d2514a6265a76645b9d6b3dc6a78777c34dbf5da9fa2d564651645685881\"},{\"ID\":\"a6f35e41-a86c-5e4d-a19a-a6c2464bfb47\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\0f21f1e28ef13030bbf0d87cbc97d1bc75f82ea53c842e9a3250a2156ced12d5\"},{\"ID\":\"4f624ca7-2c6d-5c42-b73f-be6e6baf2530\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\4d9e2ad969fbd74fd58c98b5ab61e55a523087910da5200920e2b6f641d00c67\"},{\"ID\":\"88e360ff-32af-521d-9a3f-3760c12b35e2\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\9e16a3d53d3e9b33344a6f0d4ed34c8a46448ee7636b672b61718225b8165e6e\"},{\"ID\":\"20f1a4e0-a6f3-5db3-9bf2-01fd3e9e855a\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\7eec7f59f9adce38cc0a6755da58a3589287d920d37414b5b21b5b543d910461\"},{\"ID\":\"c2b3d728-4879-5343-a92a-b735752a4724\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\8ed04b60acc0f65f541292a9e598d5f73019c8db425f8d49ea5819eab16a42f3\"},{\"ID\":\"2973e760-dc59-5800-a3de-ab9d93be81e5\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\cc71305d45f09ce377ef497f28c3a74ee027c27f20657d2c4a5f157d2457cc75\"},{\"ID\":\"454a7d36-038c-5364-8a25-fa84091869d6\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\9e8cde1ce8f5de861a5f22841f1ab9bc53d5f606d06efeacf5177f340e8d54d0\"},{\"ID\":\"9b748c8c-69eb-55fb-a1c1-5688cac4efd8\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\8e02bf5404057055a71d542780a2bb2731be4b3707c01918ba969fb4d83b98ec\"},{\"ID\":\"bfde5c26-b33f-5424-9405-9d69c2fea4d0\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\77483cedfb0964008c33d92d306734e1fab3b5e1ebb27e898f58ccfd108108f2\"},{\"ID\":\"bdabfbf5-80d1-57f1-86f3-448ce97e2d05\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\aed2ebbb31ad24b38ee8521dd17744319f83d267bf7f360bc177e27ae9a006cf\"},{\"ID\":\"ad9b34f2-dcee-59ea-8962-b30704ae6331\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\d44d3a675fec1070b61d6ea9bacef4ac12513caf16bd6453f043eed2792f75d8\"}],\"HostName\":\"fabrikamfiber.web-789699744-rqv6p\",\"MappedDirectories\":[{\"HostPath\":\"c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\b50f0027-ad29-11e7-b16e-000d3afd2878\\\\volumes\\\\kubernetes.io~secret\\\\default-token-rw9dn\",\"ContainerPath\":\"c:\\\\var\\\\run\\\\secrets\\\\kubernetes.io\\\\serviceaccount\",\"ReadOnly\":true,\"BandwidthMaximum\":0,\"IOPSMaximum\":0}],\"HvPartition\":false,\"EndpointList\":null,\"NetworkSharedContainerName\":\"586870f5833279678773cb700db3c175afc81d557a75867bf39b43f985133d13\",\"Servicing\":false,\"AllowUnqualifiedDNSQuery\":false}"}
33m 11s 151 kubelet, 38519acs9011 Warning FailedSync Error syncing pod
32m 11s 139 kubelet, 38519acs9011 spec.containers{fabrikamfiberweb} Warning BackOff Back-off restarting failed container
Mitigation - Using Node Labels & NodeSelector
kubectl get node will get a list of all nodes, then you can use kubectl describe node <nodename> to get more details.
In this case, there are two Windows nodes running different versions:
$ kubectl get node
NAME STATUS AGE VERSION
38519acs9010 Ready 21h v1.7.7-7+e79c96c8ff2d8e
38519acs9011 Ready 4h v1.7.7-25+bc3094f1d650a2
k8s-linuxpool1-38519084-0 Ready 21h v1.7.7
k8s-master-38519084-0 Ready 21h v1.7.7
$ kubectl describe node 38519acs9010
Name: 38519acs9010
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_D2_v2
beta.kubernetes.io/os=windows
failure-domain.beta.kubernetes.io/region=westus2
failure-domain.beta.kubernetes.io/zone=0
kubernetes.io/hostname=38519acs9010
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Fri, 06 Oct 2017 01:41:02 +0000
...
System Info:
Machine ID: 38519acs9010
System UUID:
Boot ID:
Kernel Version: 10.0 14393 (14393.1715.amd64fre.rs1_release_inmarket.170906-1810)
OS Image:
Operating System: windows
Architecture: amd64
...
$ kubectl describe node 38519acs9011
Name: 38519acs9011
Role:
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/instance-type=Standard_DS1_v2
beta.kubernetes.io/os=windows
failure-domain.beta.kubernetes.io/region=westus2
failure-domain.beta.kubernetes.io/zone=0
kubernetes.io/hostname=38519acs9011
Annotations: node.alpha.kubernetes.io/ttl=0
volumes.kubernetes.io/controller-managed-attach-detach=true
Taints: <none>
CreationTimestamp: Fri, 06 Oct 2017 18:13:25 +0000
Conditions:
...
System Info:
Machine ID: 38519acs9011
System UUID:
Boot ID:
Kernel Version: 10.0 16299 (16299.0.amd64fre.rs3_release.170922-1354)
OS Image:
Operating System: windows
Architecture: amd64
...
- Take note of each node name, and the
Kernel Versionfrom system info:
| Name | Version |
|---|---|
| 38519acs9010 | 14393.1715.amd64fre.rs1_release_inmarket.170906-1810 |
| 38519acs9011 | 16299.0.amd64fre.rs3_release.170922-1354 |
- Add a label to each node called
beta.kubernetes.io/osbuild. Windows Server 2016 needs both major & minor (14393.1715) to be supported without Hyper-V isolation. Windows Server version 1709 only needs major (16299) to match.
From the sample above:
$ kubectl label node 38519acs9010 beta.kubernetes.io/osbuild=14393.1715
node "38519acs9010" labeled
$ kubectl label node 38519acs9011 beta.kubernetes.io/osbuild=16299
node "38519acs9011" labeled
- Check the labels are there with
kubectl get nodes --show-labels
$ kubectl get nodes --show-labels
NAME STATUS AGE VERSION LABELS
38519acs9010 Ready,SchedulingDisabled 3d v1.7.7-7+e79c96c8ff2d8e beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=windows,beta.kubernetes.io/osbuild=14393.1715,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=38519acs9010
38519acs9011 Ready 3d v1.7.7-25+bc3094f1d650a2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_DS1_v2,beta.kubernetes.io/os=windows,beta.kubernetes.io/osbuild=16299,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=38519acs9011
k8s-linuxpool1-38519084-0 Ready 3d v1.7.7 agentpool=linuxpool1,beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=k8s-linuxpool1-38519084-0,kubernetes.io/role=agent
k8s-master-38519084-0 Ready 3d v1.7.7 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=Standard_D2_v2,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=westus2,failure-domain.beta.kubernetes.io/zone=0,kubernetes.io/hostname=k8s-master-38519084-0,kubernetes.io/role=master
- Add node selectors to deployments
Add a nodeSelector to the container spec with beta.kubernetes.io/os = windows and beta.kubernetes.io/osbuild = 14393.* or 16299 to match the base OS used by the container.
Here's a full example for running a container built for Windows Server 2016:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
kompose.cmd: kompose convert -f docker-compose-combined.yml
kompose.version: 1.2.0 (99f88ef)
creationTimestamp: null
labels:
io.kompose.service: fabrikamfiber.web
name: fabrikamfiber.web
spec:
replicas: 1
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
io.kompose.service: fabrikamfiber.web
spec:
containers:
- image: patricklang/fabrikamfiber.web:latest
name: fabrikamfiberweb
ports:
- containerPort: 80
resources: {}
restartPolicy: Always
nodeSelector:
"beta.kubernetes.io/os": windows
"beta.kubernetes.io/osbuild": "14393.1715"
status: {}
The pod can now start with the updated deployment. The node selectors are also shown in kubectl describe pod <podname> so you can verify they were added.
$ kubectl -n plang describe po fa
Name: fabrikamfiber.web-1780117715-5c8vw
Namespace: plang
Node: 38519acs9010/10.240.0.4
Start Time: Tue, 10 Oct 2017 01:43:28 +0000
Labels: io.kompose.service=fabrikamfiber.web
pod-template-hash=1780117715
Annotations: kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"plang","name":"fabrikamfiber.web-1780117715","uid":"6a07aaf3-ad5c-11e7-b16e-000d3...
Status: Running
IP: 10.244.1.84
Created By: ReplicaSet/fabrikamfiber.web-1780117715
Controlled By: ReplicaSet/fabrikamfiber.web-1780117715
Containers:
fabrikamfiberweb:
Container ID: docker://c94594fb53161f3821cf050d9af7546991aaafbeab41d333d9f64291327fae13
Image: patricklang/fabrikamfiber.web:latest
Image ID: docker-pullable://patricklang/fabrikamfiber.web@sha256:562741016ce7d9a232a389449a4fd0a0a55aab178cf324144404812887250ead
Port: 80/TCP
State: Running
Started: Tue, 10 Oct 2017 01:43:42 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rw9dn (ro)
Conditions:
Type Status
Initialized True
Ready True
PodScheduled True
Volumes:
default-token-rw9dn:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rw9dn
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=windows
beta.kubernetes.io/osbuild=14393.1715
Tolerations: <none>
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
5m 5m 1 default-scheduler Normal Scheduled Successfully assigned fabrikamfiber.web-1780117715-5c8vw to 38519acs9010
5m 5m 1 kubelet, 38519acs9010 Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "default-token-rw9dn"
5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Pulling pulling image "patricklang/fabrikamfiber.web:latest"
5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Pulled Successfully pulled image "patricklang/fabrikamfiber.web:latest"
5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Created Created container
5m 5m 1 kubelet, 38519acs9010 spec.containers{fabrikamfiberweb} Normal Started Started container