Configure group Managed Service Accounts (gMSA) for Windows containers with Azure Kubernetes Service on Azure Stack HCI and Windows Server

Article
04/30/2024

Applies to: AKS on Azure Stack HCI 22H2, AKS on Windows Server

To use AD Authentication, you can configure group Managed Service Accounts (gMSA) for Windows containers to run with a non-domain joined host. A group Managed Service Account is a special type of service account introduced in Windows Server 2012 that's designed to allow multiple computers to share an identity without knowing the password. Windows containers cannot be domain joined, but many Windows applications that run in Windows containers still need AD Authentication.

Note

To learn how the Kubernetes community supports using gMSA with Windows containers, see Configure gMSA.

Architecture of gMSA for containers with a non-domain joined host

gMSA for containers with a non-domain joined host uses a portable user identity instead of a host identity to retrieve gMSA credentials. Therefore, manually joining Windows worker nodes to a domain is no longer necessary. The user identity is saved as a secret in Kubernetes. The following diagram shows the process for configuring gMSA for containers with a non-domain joined host:

Diagram of group Managed Service Accounts version two

gMSA for containers with a non-domain joined host provides the flexibility of creating containers with gMSA without joining the host node to the domain. Starting with Windows Server 2019, ccg.exe is supported, which enables a plug-in mechanism to retrieve gMSA credentials from Active Directory. You can use that identity to start the container. For more information on ccg.exe, see ICcgDomainAuthCredentials interface.

Comparison of gMSA for containers with a non-domain joined host and a domain joined host

When gMSA was initially introduced, it required the container host to be domain joined, which created a lot of overhead to join Windows worker nodes manually to a domain. This limitation was addressed with gMSA for containers with a non-domain joined host, so users can now use gMSA with domain-unjoined hosts. Other improvements to gMSA include the following features:

Eliminating the requirement to manually join Windows worker nodes to a domain, which caused a lot of overhead. For scaling scenarios, this simplifies the process.
In rolling update scenarios, users no longer need to rejoin the node to a domain.
An easier process for managing the worker node machine accounts to retrieve gMSA service account passwords.
A less complicated end-to-end process to configure gMSA with Kubernetes.

Before you begin

To run a Windows container with a group managed service account, you need the following prerequisites:

An Active Directory domain with at least one domain controller running Windows Server 2012 or later. There are no forest or domain functional level requirements to use gMSAs, but only domain controllers running Windows Server 2012 or later can distribute gMSA passwords. For more information, see Active Directory requirements for gMSAs.
Permission to create a gMSA account. To create a gMSA account, you must be a Domain Administrator or use an account that has permissions to create msDS-GroupManagedServiceAccount objects.
Access to the internet to download the CredentialSpec PowerShell module. If you're working in a disconnected environment, you can save the module on a computer with internet access and copy it to your development machine or container host.
To ensure gMSA and AD authentication work, ensure that the Azure Stack HCI and Windows Server cluster nodes are configured to synchronize their time with either a domain controller or other time source. You should also make sure Hyper-V is configured to synchronize time to any virtual machines.

Prepare the gMSA in the domain controller

Follow these steps to prepare the gMSA in the domain controller:

In the domain controller, prepare Active Directory and create the gMSA account.
Create a domain user account. This user account/password credentials are saved as a Kubernetes secret and used to retrieve the gMSA password.

To create a GMSA account and grant permission to read the password for the gMSA account created in Step 2, run the following New-ADServiceAccount PowerShell command:

 New-ADServiceAccount -Name "<gmsa account name>" -DnsHostName "<gmsa account name>.<domain name>.com" -ServicePrincipalNames "host/<gmsa account name>", "host/<gmsa account name>.<domain name>.com" -PrincipalsAllowedToRetrieveManagedPassword <username you created earlier>

For -PrincipalsAllowedToRetrieveManagedPassword, specify the domain username you created earlier as shown in the following example:

New-ADServiceAccount -Name "WebApp01" -DnsHostName "WebApp01.akshcitest.com" -ServicePrincipalNames "host/WebApp01", "host/WebApp01.akshcitest.com" -PrincipalsAllowedToRetrieveManagedPassword "testgmsa"

Prepare the gMSA credential spec JSON file

To prepare the gMSA credential spec JSON file, follow the steps for creating a credential spec.

Configure gMSA for Windows pods and containers in the cluster

The Kubernetes community already supports domain joined Windows worker nodes for gMSA. Although you don't need to domain join a Windows worker node in AKS on Azure Stack HCI and Windows Server, there are other required configuration steps. These steps include installing the webhook, the custom resource definition (CRD), and the credential spec, and enabling role-based access control (RBAC role). The following steps use PowerShell commands that are built to help simplify the configuration process.

Before completing the following steps, make sure the AksHci PowerShell module is installed and kubectl can connect to your cluster.

To install the webhook, run the following Install-AksHciGmsaWebhook PowerShell command:
```
Install-AksHciGMSAWebhook -Name <cluster name>
```
To validate that the webhook pod is successfully running, run the following command:
```
kubectl get pods -n kube-system | findstr gmsa
```
You should see one pod with the gmsa-webhook prefix that is running.
Create the secret object that stores the Active Directory user credential. Complete the following configuration data and save the settings into a file named secret.yaml.
```
apiVersion: v1
kind: Secret
metadata:
   name: <secret-name>
   namespace: <secret under namespace other than the default>
type: Opaque
stringData:
   domain: <FQDN of the domain, for example: akshcitest.com>
   username: <domain user who can retrieve the gMSA password>
   password: <password>
```
Then, run the following command to create the secret object:
```
kubectl apply -f secret.yaml
```
Note

If you create a secret under a namespace other than the default, remember to set the namespace of the deployment to the same namespace.

Use the Add-AksHciGMSACredentialSpec PowerShell cmdlet to create the gMSA CRD, enable role-based access control (RBAC), and then assign the role to the service accounts to use a specific gMSA credential spec file. These steps are described in more detail in this Kubernetes article on Configure gMSA for Windows pods and containers.

Use the JSON credential spec as input for the following PowerShell command (parameters with asterisks * are mandatory):

Add-AksHciGMSACredentialSpec -Name <cluster name>*  
  -credSpecFilePath <path to JSON credspec>* 
  -credSpecName <name for credspec as the k8s GMSACredentialSpec object>* 
  -secretName <name of secret>* 
  -secretNamespace <namespace of secret>  
  -serviceAccount <name of service account to bind to clusterrole>  
  -clusterRoleName <name of clusterrole to use the credspec>*  
  -overwrite

To view an example, see the following code:

Add-AksHciGMSACredentialSpec -Name mynewcluster 
  -credSpecFilePath .\credspectest.json 
  -credSpecName credspec-mynewcluster 
  -secretName mysecret 
  -clusterRoleName clusterrole-mynewcluster

Deploy the application

Create the deployment YAML file using the following example settings. The required entries include serviceAccountName, gmsaCredentialSpecName, volumeMounts, and dnsconfig.

Add the service account:

serviceAccountName: default
   securityContext: 
     windowsOptions: 
       gmsaCredentialSpecName:

Add the credential spec object:

securityContext: 
     windowsOptions: 
       gmsaCredentialSpecName: <cred spec name>

Mount the secret for the deployment:

volumeMounts:
     - name: <volume name>
       mountPath: <vmount path>
       readOnly: true
   volumes:
     - name: <volume name>
       secret:
         secretName: <secret name>
         items:
           - key: username
             path: my-group/my-username

Add the IP address of the domain controller and domain name under dnsConfig:

dnsConfig: 
     nameservers:
       - <IP address for domain controller>
     searches: 
       - <domain>

Verify the container is working with gMSA

After you deploy the container, use the following steps to verify that it's working:

Get the container ID for your deployment by running the following command:
```
kubectl get pods
```
Open PowerShell and run the following command:
```
kubectl exec -it <container id> powershell
```
Once you are in the container, run the following command:
```
Nltest /parentdomain 
Nltest /sc_verify:<domain> 
```
```
Connection Status = 0 0x0 NERR_Success The command completed successfully. 
```
This output shows that the computer was authenticated by a domain controller, and a secure channel exists between the client computer and the domain controller.
Check if the container can obtain a valid Kerberos Ticket Granting Ticket (TGT).
```
klist get krbtgt
```
```
A ticket to krbtgt has been retrieved successfully
```

Clean up gMSA settings in the cluster

If you need to clean up gMSA settings, use the following uninstall steps.

Uninstall the credential spec

To uninstall, run the following Remove-AksHcigmsaCredentialSpec PowerShell command:

Remove-AksHciGMSACredentialSpec -Name <cluster name> -credSpecName <cred spec object name> -clusterRoleName <clusterrole object name> -serviceAccount <serviceaccount object name> -secretNamespace <namespace of the secret object>

Uninstall webhook

To uninstall the webhook, run the following Uninstall-AksHciGMSAWebhook PowerShell command:

Uninstall-AksHciGMSAWebhook -Name <cluster name>