How to query logs from Azure Monitor for containers
- 4 minutes to read
Azure Monitor for containers collects performance metrics, inventory data, and health state information from container hosts and containers. The data is collected every three minutes and forwarded to the Log Analytics workspace in Azure Monitor. This data is available for query in Azure Monitor. You can apply this data to scenarios that include migration planning, capacity analysis, discovery, and on-demand performance troubleshooting.
Container records
In the following table, details of the records collected by Azure Monitor for containers are provided.
| Data | Data source | Data type | Fields |
|---|---|---|---|
| Performance for hosts and containers | Usage metrics are obtained from cAdvisor and limits from Kube api | Perf |
Computer, ObjectName, CounterName (%Processor Time, Disk Reads MB, Disk Writes MB, Memory Usage MB, Network Receive Bytes, Network Send Bytes, Processor Usage sec, Network), CounterValue, TimeGenerated, CounterPath, SourceSystem |
| Container inventory | Docker | ContainerInventory |
TimeGenerated, Computer, container name, ContainerHostname, Image, ImageTag, ContainerState, ExitCode, EnvironmentVar, Command, CreatedTime, StartedTime, FinishedTime, SourceSystem, ContainerID, ImageID |
| Container log | Docker | ContainerLog |
TimeGenerated, Computer, image ID, container name, LogEntrySource, LogEntry, SourceSystem, ContainerID |
| Container node inventory | Kube API | ContainerNodeInventory |
TimeGenerated, Computer, ClassName_s, DockerVersion_s, OperatingSystem_s, Volume_s, Network_s, NodeRole_s, OrchestratorType_s, InstanceID_g, SourceSystem |
| Inventory of pods in a Kubernetes cluster | Kube API | KubePodInventory |
TimeGenerated, Computer, ClusterId, ContainerCreationTimeStamp, PodUid, PodCreationTimeStamp, ContainerRestartCount, PodRestartCount, PodStartTime, ContainerStartTime, ServiceName, ControllerKind, ControllerName, ContainerStatus, ContainerStatusReason, ContainerID, ContainerName, Name, PodLabel, Namespace, PodStatus, ClusterName, PodIp, SourceSystem |
| Inventory of nodes part of a Kubernetes cluster | Kube API | KubeNodeInventory |
TimeGenerated, Computer, ClusterName, ClusterId, LastTransitionTimeReady, Labels, Status, KubeletVersion, KubeProxyVersion, CreationTimeStamp, SourceSystem |
| Kubernetes Events | Kube API | KubeEvents |
TimeGenerated, Computer, ClusterId_s, FirstSeen_t, LastSeen_t, Count_d, ObjectKind_s, Namespace_s, Name_s, Reason_s, Type_s, TimeGenerated_s, SourceComponent_s, ClusterName_s, Message, SourceSystem |
| Services in the Kubernetes cluster | Kube API | KubeServices |
TimeGenerated, ServiceName_s, Namespace_s, SelectorLabels_s, ClusterId_s, ClusterName_s, ClusterIP_s, ServiceType_s, SourceSystem |
| Performance metrics for nodes part of the Kubernetes cluster | Perf | where ObjectName == "K8SNode" | Computer, ObjectName, CounterName (cpuAllocatableBytes, memoryAllocatableBytes, cpuCapacityNanoCores, memoryCapacityBytes, memoryRssBytes, cpuUsageNanoCores, memoryWorkingsetBytes, restartTimeEpoch), CounterValue, TimeGenerated, CounterPath, SourceSystem | |
| Performance metrics for containers part of the Kubernetes cluster | Perf | where ObjectName == "K8SContainer" | CounterName ( cpuRequestNanoCores, memoryRequestBytes, cpuLimitNanoCores, memoryWorkingSetBytes, restartTimeEpoch, cpuUsageNanoCores, memoryRssBytes), CounterValue, TimeGenerated, CounterPath, SourceSystem | |
| Custom Metrics | InsightsMetrics |
Computer, Name, Namespace, Origin, SourceSystem, Tags1, TimeGenerated, Type, Va, _ResourceId |
1 The Tags property represents multiple dimensions for the corresponding metric. For more information about the metrics collected and stored in the InsightsMetrics table and a description of the record properties, see InsightsMetrics overview.
Search logs to analyze data
Azure Monitor Logs can help you look for trends, diagnose bottlenecks, forecast, or correlate data that can help you determine whether the current cluster configuration is performing optimally. Pre-defined log searches are provided for you to immediately start using or to customize to return the information the way you want.
You can perform interactive analysis of data in the workspace by selecting the View Kubernetes event logs or View container logs option in the preview pane from the View in analytics drop-down list. The Log Search page appears to the right of the Azure portal page that you were on.
The container logs output that's forwarded to your workspace are STDOUT and STDERR. Because Azure Monitor is monitoring Azure-managed Kubernetes (AKS), Kube-system is not collected today because of the large volume of generated data.
Example log search queries
It's often useful to build queries that start with an example or two and then modify them to fit your requirements. To help build more advanced queries, you can experiment with the following sample queries:
| Query | Description |
|---|---|
| ContainerInventory | project Computer, Name, Image, ImageTag, ContainerState, CreatedTime, StartedTime, FinishedTime | render table |
List all of a container's lifecycle information |
| KubeEvents_CL | where not(isempty(Namespace_s)) | sort by TimeGenerated desc | render table |
Kubernetes events |
| ContainerImageInventory | summarize AggregatedValue = count() by Image, ImageTag, Running |
Image inventory |
| Select the Line chart display option: Perf | where ObjectName == "K8SContainer" and CounterName == "cpuUsageNanoCores" | summarize AvgCPUUsageNanoCores = avg(CounterValue) by bin(TimeGenerated, 30m), InstanceName |
Container CPU |
| Select the Line chart display option: Perf | where ObjectName == "K8SContainer" and CounterName == "memoryRssBytes" | summarize AvgUsedRssMemoryBytes = avg(CounterValue) by bin(TimeGenerated, 30m), InstanceName |
Container memory |
| InsightsMetrics | where Name == "requests_count" | summarize Val=any(Val) by TimeGenerated=bin(TimeGenerated, 1m) | sort by TimeGenerated asc | project RequestsPerMinute = Val - prev(Val), TimeGenerated | render barchart |
Requests Per Minute with Custom Metrics |
Query Prometheus metrics data
The following example is a Prometheus metrics query showing disk reads per second per disk per node.
InsightsMetrics
| where Namespace == 'container.azm.ms/diskio'
| where TimeGenerated > ago(1h)
| where Name == 'reads'
| extend Tags = todynamic(Tags)
| extend HostName = tostring(Tags.hostName), Device = Tags.name
| extend NodeDisk = strcat(Device, "/", HostName)
| order by NodeDisk asc, TimeGenerated asc
| serialize
| extend PrevVal = iif(prev(NodeDisk) != NodeDisk, 0.0, prev(Val)), PrevTimeGenerated = iif(prev(NodeDisk) != NodeDisk, datetime(null), prev(TimeGenerated))
| where isnotnull(PrevTimeGenerated) and PrevTimeGenerated != TimeGenerated
| extend Rate = iif(PrevVal > Val, Val / (datetime_diff('Second', TimeGenerated, PrevTimeGenerated) * 1), iif(PrevVal == Val, 0.0, (Val - PrevVal) / (datetime_diff('Second', TimeGenerated, PrevTimeGenerated) * 1)))
| where isnotnull(Rate)
| project TimeGenerated, NodeDisk, Rate
| render timechart
To view Prometheus metrics scraped by Azure Monitor filtered by Namespace, specify "prometheus". Here is a sample query to view Prometheus metrics from the default kubernetes namespace.
InsightsMetrics
| where Namespace == "prometheus"
| extend tags=parse_json(Tags)
| summarize count() by Name
Prometheus data can also be directly queried by name.
InsightsMetrics
| where Namespace == "prometheus"
| where Name contains "some_prometheus_metric"
Query config or scraping errors
To investigate any configuration or scraping errors, the following example query returns informational events from the KubeMonAgentEvents table.
KubeMonAgentEvents | where Level != "Info"
The output shows results similar to the following example:
Next steps
Azure Monitor for containers does not include a predefined set of alerts. Review the Create performance alerts with Azure Monitor for containers to learn how to create recommended alerts for high CPU and memory utilization to support your DevOps or operational processes and procedures.