Understand Azure Policy for Kubernetes clusters

Azure Policy extends Gatekeeper v3, an admission controller webhook for Open Policy Agent (OPA), to apply at-scale enforcements and safeguards on your clusters in a centralized, consistent manner. Azure Policy makes it possible to manage and report on the compliance state of your Kubernetes clusters from one place. The add-on enacts the following functions:

  • Checks with Azure Policy service for policy assignments to the cluster.
  • Deploys policy definitions into the cluster as constraint template and constraint custom resources.
  • Reports auditing and compliance details back to Azure Policy service.

Azure Policy for Kubernetes supports the following cluster environments:

Important

The add-ons for AKS Engine and Arc enabled Kubernetes are in preview. Azure Policy for Kubernetes only supports Linux node pools and built-in policy definitions. Built-in policy definitions are in the Kubernetes category. The limited preview policy definitions with EnforceOPAConstraint and EnforceRegoPolicy effect and the related Kubernetes Service category are deprecated. Instead, use the effects audit and deny with Resource Provider mode Microsoft.Kubernetes.Data.

Overview

To enable and use Azure Policy with your Kubernetes cluster, take the following actions:

  1. Configure your Kubernetes cluster and install the add-on:

    Note

    For common issues with installation, see Troubleshoot - Azure Policy Add-on.

  2. Understand the Azure Policy language for Kubernetes

  3. Assign a built-in definition to your Kubernetes cluster

  4. Wait for validation

Limitations

The following general limitations apply to the Azure Policy Add-on for Kubernetes clusters:

  • Azure Policy Add-on for Kubernetes is supported on Kubernetes version 1.14 or higher.
  • Azure Policy Add-on for Kubernetes can only be deployed to Linux node pools
  • Only built-in policy definitions are supported
  • Maximum number of Non-compliant records per policy per cluster: 500
  • Maximum number of Non-compliant records per subscription: 1 million
  • Installations of Gatekeeper outside of the Azure Policy Add-on aren't supported. Uninstall any components installed by a previous Gatekeeper installation before enabling the Azure Policy Add-on.
  • Reasons for non-compliance aren't available for the Microsoft.Kubernetes.Data Resource Provider mode. Use Component details.
  • Exemptions aren't supported for Resource Provider modes.

The following limitations apply only to the Azure Policy Add-on for AKS:

  • AKS Pod security policy and the Azure Policy Add-on for AKS can't both be enabled. For more information, see AKS pod security limitation.
  • Namespaces automatically excluded by Azure Policy Add-on for evaluation: kube-system, gatekeeper-system, and aks-periscope.

Recommendations

The following are general recommendations for using the Azure Policy Add-on:

  • The Azure Policy Add-on requires 3 Gatekeeper components to run: 1 audit pod and 2 webhook pod replicas. These components consume more resources as the count of Kubernetes resources and policy assignments increases in the cluster which requires audit and enforcement operations.

    • For less than 500 pods in a single cluster with a max of 20 constraints: 2 vCPUs and 350 MB memory per component.
    • For more than 500 pods in a single cluster with a max of 40 constraints: 3 vCPUs and 600 MB memory per component.
  • Windows pods don't support security contexts. Thus, some of the Azure Policy definitions, such as disallowing root privileges, can't be escalated in Windows pods and only apply to Linux pods.

The following recommendation applies only to AKS and the Azure Policy Add-on:

  • Use system node pool with CriticalAddonsOnly taint to schedule Gatekeeper pods. For more information, see Using system node pools.
  • Secure outbound traffic from your AKS clusters. For more information, see Control egress traffic for cluster nodes.
  • If the cluster has aad-pod-identity enabled, Node Managed Identity (NMI) pods modify the nodes' iptables to intercept calls to the Azure Instance Metadata endpoint. This configuration means any request made to the Metadata endpoint is intercepted by NMI even if the pod doesn't use aad-pod-identity. AzurePodIdentityException CRD can be configured to inform aad-pod-identity that any requests to the Metadata endpoint originating from a pod that matches labels defined in CRD should be proxied without any processing in NMI. The system pods with kubernetes.azure.com/managedby: aks label in kube-system namespace should be excluded in aad-pod-identity by configuring the AzurePodIdentityException CRD. For more information, see Disable aad-pod-identity for a specific pod or application. To configure an exception, install the mic-exception YAML.

Install Azure Policy Add-on for AKS

Before installing the Azure Policy Add-on or enabling any of the service features, your subscription must enable the Microsoft.ContainerService and Microsoft.PolicyInsights resource providers.

  1. You need the Azure CLI version 2.12.0 or later installed and configured. Run az --version to find the version. If you need to install or upgrade, see Install the Azure CLI.

  2. Register the resource providers and preview features.

    • Azure portal:

      Register the Microsoft.ContainerService and Microsoft.PolicyInsights resource providers. For steps, see Resource providers and types.

    • Azure CLI:

      # Log in first with az login if you're not using Cloud Shell
      
      # Provider register: Register the Azure Kubernetes Service provider
      az provider register --namespace Microsoft.ContainerService
      
      # Provider register: Register the Azure Policy provider
      az provider register --namespace Microsoft.PolicyInsights
      
  3. If limited preview policy definitions were installed, remove the add-on with the Disable button on your AKS cluster under the Policies page.

  4. The AKS cluster must be version 1.14 or higher. Use the following script to validate your AKS cluster version:

    # Log in first with az login if you're not using Cloud Shell
    
    # Look for the value in kubernetesVersion
    az aks list
    
  5. Install version 2.12.0 or higher of the Azure CLI. For more information, see Install the Azure CLI.

Once the above prerequisite steps are completed, install the Azure Policy Add-on in the AKS cluster you want to manage.

  • Azure portal

    1. Launch the AKS service in the Azure portal by selecting All services, then searching for and selecting Kubernetes services.

    2. Select one of your AKS clusters.

    3. Select Policies on the left side of the Kubernetes service page.

    4. In the main page, select the Enable add-on button.

      Note

      If the Disable add-on button is enabled and a migration warning v2 message is displayed, v1 add-on is installed and must be removed prior to assigning v2 policy definitions. The deprecated v1 add-on will automatically be replaced with the v2 add-on starting August 24, 2020. New v2 versions of the policy definitions must then be assigned. To upgrade now, follow these steps:

      1. Validate your AKS cluster has the v1 add-on installed by visiting the Policies page on your AKS cluster and has the "The current cluster uses Azure Policy add-on v1..." message.
      2. Remove the add-on.
      3. Select the Enable add-on button to install the v2 version of the add-on.
      4. Assign v2 versions of your v1 built-in policy definitions
  • Azure CLI

    # Log in first with az login if you're not using Cloud Shell
    
    az aks enable-addons --addons azure-policy --name MyAKSCluster --resource-group MyResourceGroup
    

To validate that the add-on installation was successful and that the azure-policy and gatekeeper pods are running, run the following command:

# azure-policy pod is installed in kube-system namespace
kubectl get pods -n kube-system

# gatekeeper pod is installed in gatekeeper-system namespace
kubectl get pods -n gatekeeper-system

Lastly, verify that the latest add-on is installed by running this Azure CLI command, replacing <rg> with your resource group name and <cluster-name> with the name of your AKS cluster: az aks show --query addonProfiles.azurepolicy -g <rg> -n <cluster-name>. The result should look similar to the following output and config.version should be v2:

"addonProfiles": {
    "azurepolicy": {
        "config": {
            "version": "v2"
        },
        "enabled": true,
        "identity": null
    },
}

Install Azure Policy Add-on for Azure Arc enabled Kubernetes (preview)

Before installing the Azure Policy Add-on or enabling any of the service features, your subscription must enable the Microsoft.PolicyInsights resource provider and create a role assignment for the cluster service principal.

  1. You need the Azure CLI version 2.12.0 or later installed and configured. Run az --version to find the version. If you need to install or upgrade, see Install the Azure CLI.

  2. To enable the resource provider, follow the steps in Resource providers and types or run either the Azure CLI or Azure PowerShell command:

    • Azure CLI

      # Log in first with az login if you're not using Cloud Shell
      
      # Provider register: Register the Azure Policy provider
      az provider register --namespace 'Microsoft.PolicyInsights'
      
    • Azure PowerShell

      # Log in first with Connect-AzAccount if you're not using Cloud Shell
      
      # Provider register: Register the Azure Policy provider
      Register-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'
      
  3. The Kubernetes cluster must be version 1.14 or higher.

  4. Install Helm 3.

  5. Your Kubernetes cluster enabled for Azure Arc. For more information, see onboarding a Kubernetes cluster to Azure Arc.

  6. Have the fully qualified Azure Resource ID of the Azure Arc enabled Kubernetes cluster.

  7. Open ports for the add-on. The Azure Policy Add-on uses these domains and ports to fetch policy definitions and assignments and report compliance of the cluster back to Azure Policy.

    Domain Port
    gov-prod-policy-data.trafficmanager.net 443
    raw.githubusercontent.com 443
    login.windows.net 443
    dc.services.visualstudio.com 443
  8. Assign 'Policy Insights Data Writer (Preview)' role assignment to the Azure Arc enabled Kubernetes cluster. Replace <subscriptionId> with your subscription ID, <rg> with the Azure Arc enabled Kubernetes cluster's resource group, and <clusterName> with the name of the Azure Arc enabled Kubernetes cluster. Keep track of the returned values for appId, password, and tenant for the installation steps.

    • Azure CLI

      az ad sp create-for-rbac --role "Policy Insights Data Writer (Preview)" --scopes "/subscriptions/<subscriptionId>/resourceGroups/<rg>/providers/Microsoft.Kubernetes/connectedClusters/<clusterName>"
      
    • Azure PowerShell

      $sp = New-AzADServicePrincipal -Role "Policy Insights Data Writer (Preview)" -Scope "/subscriptions/<subscriptionId>/resourceGroups/<rg>/providers/Microsoft.Kubernetes/connectedClusters/<clusterName>"
      
      @{ appId=$sp.ApplicationId;password=[System.Runtime.InteropServices.Marshal]::PtrToStringAuto([System.Runtime.InteropServices.Marshal]::SecureStringToBSTR($sp.Secret));tenant=(Get-AzContext).Tenant.Id } | ConvertTo-Json
      

    Sample output of the above commands:

    {
        "appId": "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa",
        "password": "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb",
        "tenant": "cccccccc-cccc-cccc-cccc-cccccccccccc"
    }
    

Once the above prerequisite steps are completed, install the Azure Policy Add-on in your Azure Arc enabled Kubernetes cluster:

  1. Add the Azure Policy Add-on repo to Helm:

    helm repo add azure-policy https://raw.githubusercontent.com/Azure/azure-policy/master/extensions/policy-addon-kubernetes/helm-charts
    
  2. Install the Azure Policy Add-on using Helm Chart:

    # In below command, replace the following values with those gathered above.
    #    <AzureArcClusterResourceId> with your Azure Arc enabled Kubernetes cluster resource Id. For example: /subscriptions/<subscriptionId>/resourceGroups/<rg>/providers/Microsoft.Kubernetes/connectedClusters/<clusterName>
    #    <ServicePrincipalAppId> with app Id of the service principal created during prerequisites.
    #    <ServicePrincipalPassword> with password of the service principal created during prerequisites.
    #    <ServicePrincipalTenantId> with tenant of the service principal created during prerequisites.
    helm install azure-policy-addon azure-policy/azure-policy-addon-arc-clusters \
        --set azurepolicy.env.resourceid=<AzureArcClusterResourceId> \
        --set azurepolicy.env.clientid=<ServicePrincipalAppId> \
        --set azurepolicy.env.clientsecret=<ServicePrincipalPassword> \
        --set azurepolicy.env.tenantid=<ServicePrincipalTenantId>
    

    For more information about what the add-on Helm Chart installs, see the Azure Policy Add-on Helm Chart definition on GitHub.

To validate that the add-on installation was successful and that the azure-policy and gatekeeper pods are running, run the following command:

# azure-policy pod is installed in kube-system namespace
kubectl get pods -n kube-system

# gatekeeper pod is installed in gatekeeper-system namespace
kubectl get pods -n gatekeeper-system

Install Azure Policy Add-on for AKS Engine (preview)

Before installing the Azure Policy Add-on or enabling any of the service features, your subscription must enable the Microsoft.PolicyInsights resource provider and create a role assignment for the cluster service principal.

  1. You need the Azure CLI version 2.0.62 or later installed and configured. Run az --version to find the version. If you need to install or upgrade, see Install the Azure CLI.

  2. To enable the resource provider, follow the steps in Resource providers and types or run either the Azure CLI or Azure PowerShell command:

    • Azure CLI

      # Log in first with az login if you're not using Cloud Shell
      
      # Provider register: Register the Azure Policy provider
      az provider register --namespace 'Microsoft.PolicyInsights'
      
    • Azure PowerShell

      # Log in first with Connect-AzAccount if you're not using Cloud Shell
      
      # Provider register: Register the Azure Policy provider
      Register-AzResourceProvider -ProviderNamespace 'Microsoft.PolicyInsights'
      
  3. Create a role assignment for the cluster service principal.

    • If you don't know the cluster service principal app ID, look it up with the following command.

      # Get the kube-apiserver pod name
      kubectl get pods -n kube-system
      
      # Find the aadClientID value
      kubectl exec <kube-apiserver pod name> -n kube-system cat /etc/kubernetes/azure.json
      
    • Assign 'Policy Insights Data Writer (Preview)' role assignment to the cluster service principal app ID (value aadClientID from previous step) with Azure CLI. Replace <subscriptionId> with your subscription ID and <aks engine cluster resource group> with the resource group the AKS Engine self-managed Kubernetes cluster is in.

      az role assignment create --assignee <cluster service principal app ID> --scope "/subscriptions/<subscriptionId>/resourceGroups/<aks engine cluster resource group>" --role "Policy Insights Data Writer (Preview)"
      

Once the above prerequisite steps are completed, install the Azure Policy Add-on. The installation can be during the creation or update cycle of an AKS Engine or as an independent action on an existing cluster.

  • Install during creation or update cycle

    To enable the Azure Policy Add-on during the creation of a new self-managed cluster or as an update to an existing cluster, include the addons property cluster definition for AKS Engine.

    "addons": [{
        "name": "azure-policy",
        "enabled": true
    }]
    

    For more information about, see the external guide AKS Engine cluster definition.

  • Install in existing cluster with Helm Charts

    Use the following steps to prepare the cluster and install the add-on:

    1. Install Helm 3.

    2. Add the Azure Policy repo to Helm.

      helm repo add azure-policy https://raw.githubusercontent.com/Azure/azure-policy/master/extensions/policy-addon-kubernetes/helm-charts
      

      For more information, see Helm Chart - Quickstart Guide.

    3. Install the add-on with a Helm Chart. Replace <subscriptionId> with your subscription ID and <aks engine cluster resource group> with the resource group the AKS Engine self-managed Kubernetes cluster is in.

      helm install azure-policy-addon azure-policy/azure-policy-addon-aks-engine --set azurepolicy.env.resourceid="/subscriptions/<subscriptionId>/resourceGroups/<aks engine cluster resource group>"
      

      For more information about what the add-on Helm Chart installs, see the Azure Policy Add-on Helm Chart definition on GitHub.

      Note

      Because of the relationship between Azure Policy Add-on and the resource group id, Azure Policy supports only one AKS Engine cluster for each resource group.

To validate that the add-on installation was successful and that the azure-policy and gatekeeper pods are running, run the following command:

# azure-policy pod is installed in kube-system namespace
kubectl get pods -n kube-system

# gatekeeper pod is installed in gatekeeper-system namespace
kubectl get pods -n gatekeeper-system

Policy language

The Azure Policy language structure for managing Kubernetes follows that of existing policy definitions. With a Resource Provider mode of Microsoft.Kubernetes.Data, the effects audit and deny are used to manage your Kubernetes clusters. Audit and deny must provide details properties specific to working with OPA Constraint Framework and Gatekeeper v3.

As part of the details.constraintTemplate and details.constraint properties in the policy definition, Azure Policy passes the URIs of these CustomResourceDefinitions (CRD) to the add-on. Rego is the language that OPA and Gatekeeper support to validate a request to the Kubernetes cluster. By supporting an existing standard for Kubernetes management, Azure Policy makes it possible to reuse existing rules and pair them with Azure Policy for a unified cloud compliance reporting experience. For more information, see What is Rego?.

Assign a built-in policy definition

To assign a policy definition to your Kubernetes cluster, you must be assigned the appropriate Azure role-based access control (Azure RBAC) policy assignment operations. The Azure built-in roles Resource Policy Contributor and Owner have these operations. To learn more, see Azure RBAC permissions in Azure Policy.

Find the built-in policy definitions for managing your cluster using the Azure portal with the following steps:

  1. Start the Azure Policy service in the Azure portal. Select All services in the left pane and then search for and select Policy.

  2. In the left pane of the Azure Policy page, select Definitions.

  3. From the Category drop-down list box, use Select all to clear the filter and then select Kubernetes.

  4. Select the policy definition, then select the Assign button.

  5. Set the Scope to the management group, subscription, or resource group of the Kubernetes cluster where the policy assignment will apply.

    Note

    When assigning the Azure Policy for Kubernetes definition, the Scope must include the cluster resource. For an AKS Engine cluster, the Scope must be the resource group of the cluster.

  6. Give the policy assignment a Name and Description that you can use to identify it easily.

  7. Set the Policy enforcement to one of the values below.

    • Enabled - Enforce the policy on the cluster. Kubernetes admission requests with violations are denied.

    • Disabled - Don't enforce the policy on the cluster. Kubernetes admission requests with violations aren't denied. Compliance assessment results are still available. When rolling out new policy definitions to running clusters, Disabled option is helpful for testing the policy definition as admission requests with violations aren't denied.

  8. Select Next.

  9. Set parameter values

    • To exclude Kubernetes namespaces from policy evaluation, specify the list of namespaces in parameter Namespace exclusions. It's recommended to exclude: kube-system, gatekeeper-system, and azure-arc.
  10. Select Review + create.

Alternately, use the Assign a policy - Portal quickstart to find and assign a Kubernetes policy. Search for a Kubernetes policy definition instead of the sample 'audit vms'.

Important

Built-in policy definitions are available for Kubernetes clusters in category Kubernetes. For a list of built-in policy definitions, see Kubernetes samples.

Policy evaluation

The add-on checks in with Azure Policy service for changes in policy assignments every 15 minutes. During this refresh cycle, the add-on checks for changes. These changes trigger creates, updates, or deletes of the constraint templates and constraints.

In a Kubernetes cluster, if a namespace has either of the following labels, the admission requests with violations aren't denied. Compliance assessment results are still available.

  • control-plane
  • admission.policy.azure.com/ignore

Note

While a cluster admin may have permission to create and update constraint templates and constraints resources install by the Azure Policy Add-on, these aren't supported scenarios as manual updates are overwritten. Gatekeeper continues to evaluate policies that existed prior to installing the add-on and assigning Azure Policy policy definitions.

Every 15 minutes, the add-on calls for a full scan of the cluster. After gathering details of the full scan and any real-time evaluations by Gatekeeper of attempted changes to the cluster, the add-on reports the results back to Azure Policy for inclusion in compliance details like any Azure Policy assignment. Only results for active policy assignments are returned during the audit cycle. Audit results can also be seen as violations listed in the status field of the failed constraint. For details on Non-compliant resources, see Component details for Resource Provider modes.

Note

Each compliance report in Azure Policy for your Kubernetes clusters include all violations within the last 45 minutes. The timestamp indicates when a violation occurred.

Logging

As a Kubernetes controller/container, both the the azure-policy and gatekeeper pods keep logs in the Kubernetes cluster. The logs can be exposed in the Insights page of the Kubernetes cluster. For more information, see Monitor your Kubernetes cluster performance with Azure Monitor for containers.

To view the add-on logs, use kubectl:

# Get the azure-policy pod name installed in kube-system namespace
kubectl logs <azure-policy pod name> -n kube-system

# Get the gatekeeper pod name installed in gatekeeper-system namespace
kubectl logs <gatekeeper pod name> -n gatekeeper-system

For more information, see Debugging Gatekeeper in the Gatekeeper documentation.

Remove the add-on

Remove the add-on from AKS

To remove the Azure Policy Add-on from your AKS cluster, use either the Azure portal or Azure CLI:

  • Azure portal

    1. Launch the AKS service in the Azure portal by selecting All services, then searching for and selecting Kubernetes services.

    2. Select your AKS cluster where you want to disable the Azure Policy Add-on.

    3. Select Policies on the left side of the Kubernetes service page.

    4. In the main page, select the Disable add-on button.

  • Azure CLI

    # Log in first with az login if you're not using Cloud Shell
    
    az aks disable-addons --addons azure-policy --name MyAKSCluster --resource-group MyResourceGroup
    

Remove the add-on from Azure Arc enabled Kubernetes

To remove the Azure Policy Add-on and Gatekeeper from your Azure Arc enabled Kubernetes cluster, run the following Helm command:

helm uninstall azure-policy-addon

Remove the add-on from AKS Engine

To remove the Azure Policy Add-on and Gatekeeper from your AKS Engine cluster, use the method that aligns with how the add-on was installed:

  • If installed by setting the addons property in the cluster definition for AKS Engine:

    Redeploy the cluster definition to AKS Engine after changing the addons property for azure-policy to false:

    "addons": [{
        "name": "azure-policy",
        "enabled": false
    }]
    

    For more information, see AKS Engine - Disable Azure Policy Add-on.

  • If installed with Helm Charts, run the following Helm command:

    helm uninstall azure-policy-addon
    

Diagnostic data collected by Azure Policy Add-on

The Azure Policy Add-on for Kubernetes collects limited cluster diagnostic data. This diagnostic data is vital technical data related to software and performance. It's used in the following ways:

  • Keep Azure Policy Add-on up to date
  • Keep Azure Policy Add-on secure, reliable, performant
  • Improve Azure Policy Add-on - through the aggregate analysis of the use of the add-on

The information collected by the add-on isn't personal data. The following details are currently collected:

  • Azure Policy Add-on agent version
  • Cluster type
  • Cluster region
  • Cluster resource group
  • Cluster resource ID
  • Cluster subscription ID
  • Cluster OS (Example: Linux)
  • Cluster city (Example: Seattle)
  • Cluster state or province (Example: Washington)
  • Cluster country or region (Example: United States)
  • Exceptions/errors encountered by Azure Policy Add-on during agent installation on policy evaluation
  • Number of Gatekeeper policy definitions not installed by Azure Policy Add-on

Next steps