Enable and review Kubernetes master node logs in Azure Kubernetes Service (AKS)
With Azure Kubernetes Service (AKS), the master components such as the kube-apiserver and kube-controller-manager are provided as a managed service. You create and manage the nodes that run the kubelet and container runtime, and deploy your applications through the managed Kubernetes API server. To help troubleshoot your application and services, you may need to view the logs generated by these master components. This article shows you how to use Azure Monitor logs to enable and query the logs from the Kubernetes master components.
Before you begin
This article requires an existing AKS cluster running in your Azure account. If you do not already have an AKS cluster, create one using the Azure CLI or Azure portal. Azure Monitor logs works with both Kubernetes RBAC, Azure RBAC, and non-RBAC enabled AKS clusters.
Enable resource logs
To help collect and review data from multiple sources, Azure Monitor logs provides a query language and analytics engine that provides insights to your environment. A workspace is used to collate and analyze the data, and can integrate with other Azure services such as Application Insights and Security Center. To use a different platform to analyze the logs, you can instead choose to send resource logs to an Azure storage account or event hub. For more information, see What is Azure Monitor logs?.
Azure Monitor logs are enabled and managed in the Azure portal. To enable log collection for the Kubernetes master components in your AKS cluster, open the Azure portal in a web browser and complete the following steps:
- Select the resource group for your AKS cluster, such as myResourceGroup. Don't select the resource group that contains your individual AKS cluster resources, such as MC_myResourceGroup_myAKSCluster_eastus.
- On the left-hand side, choose Diagnostic settings.
- Select your AKS cluster, such as myAKSCluster, then choose to Add diagnostic setting.
- Enter a name, such as myAKSClusterLogs, then select the option to Send to Log Analytics.
- Select an existing workspace or create a new one. If you create a workspace, provide a workspace name, a resource group, and a location.
- In the list of available logs, select the logs you wish to enable. For this example, enable the kube-audit and kube-audit-admin logs. Common logs include the kube-apiserver, kube-controller-manager, and kube-scheduler. You can return and change the collected logs once Log Analytics workspaces are enabled.
- When ready, select Save to enable collection of the selected logs.
In addition to entries written by Kubernetes, your project's audit logs also have entries from AKS.
Audit logs are recorded into three categories: kube-audit, kube-audit-admin, and guard.
- The kube-audit category contains all audit log data for every audit event, including get, list, create, update, delete, patch, and post.
- The kube-audit-admin category is a subset of the kube-audit log category. kube-audit-admin reduces the number of logs significantly by excluding the get and list audit events from the log.
- The guard category is managed Azure AD and Azure RBAC audits. For managed Azure AD: token in, user info out. For Azure RBAC: access reviews in and out.
Schedule a test pod on the AKS cluster
To generate some logs, create a new pod in your AKS cluster. The following example YAML manifest can be used to create a basic NGINX instance. Create a file named
nginx.yaml in an editor of your choice and paste the following content:
apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: mypod image: mcr.microsoft.com/oss/nginx/nginx:1.15.5-alpine resources: requests: cpu: 100m memory: 128Mi limits: cpu: 250m memory: 256Mi ports: - containerPort: 80
Create the pod with the kubectl create command and specify your YAML file, as shown in the following example:
$ kubectl create -f nginx.yaml pod/nginx created
View collected logs
It may take up to 10 minutes for the diagnostics logs to be enabled and appear.
If you need all audit log data for compliance or other purposes, collect and store it in inexpensive storage such as blob storage. Use the kube-audit-admin log category to collect and save a meaningful set of audit log data for monitoring and alerting purposes.
In the Azure portal, navigate to your AKS cluster, and select Logs on the left-hand side. Close the Example Queries window if it appears.
On the left-hand side, choose Logs. To view the kube-audit logs, enter the following query in the text box:
AzureDiagnostics | where Category == "kube-audit" | project log_s
Many logs are likely returned. To scope down the query to view the logs about the NGINX pod created in the previous step, add an additional where statement to search for nginx as shown in the following example query:
AzureDiagnostics | where Category == "kube-audit" | where log_s contains "nginx" | project log_s
To view the kube-audit-admin logs, enter the following query in the text box:
AzureDiagnostics | where Category == "kube-audit-admin" | project log_s
In this example, the query shows all create jobs in kube-audit-admin. There are likely many results returned, to scope down the query to view the logs about the NGINX pod created in the previous step, add an additional where statement to search for nginx as shown in the following example query.
AzureDiagnostics | where Category == "kube-audit-admin" | where log_s contains "nginx" | project log_s
For more information on how to query and filter your log data, see View or analyze data collected with log analytics log search.
Log event schema
AKS logs the following events:
|aksService||The display name in audit log for the control plane operation (from the hcpService)|
|masterclient||The display name in audit log for MasterClientCertificate, the certificate you get from az aks get-credentials|
|nodeclient||The display name for ClientCertificate, which is used by agent nodes|
In this article, you learned how to enable and review the logs for the Kubernetes master components in your AKS cluster. To monitor and troubleshoot further, you can also view the Kubelet logs and enable SSH node access.