Azure logging and auditing
Azure provides a wide array of configurable security auditing and logging options to help you identify gaps in your security policies and mechanisms. This article discusses generating, collecting, and analyzing security logs from services hosted on Azure.
Certain recommendations in this article might result in increased data, network, or compute resource usage, and increase your license or subscription costs.
Types of logs in Azure
Cloud applications are complex, with many moving parts. Logs provide data to help keep your applications up and running. Logs help you troubleshoot past problems or prevent potential ones. And they can help improve application performance or maintainability, or automate actions that would otherwise require manual intervention.
Azure logs are categorized into the following types:
Control/management logs provide information about Azure Resource Manager CREATE, UPDATE, and DELETE operations. For more information, see Azure activity logs.
Data plane logs provide information about events raised as part Azure resource usage. Examples of this type of log are the Windows event system, security, and application logs in a virtual machine (VM) and the diagnostics logs that are configured through Azure Monitor.
Processed events provide information about analyzed events/alerts that have been processed on your behalf. Examples of this type are Azure Security Center alerts where Azure Security Center has processed and analyzed your subscription and provides concise security alerts.
The following table lists the most important types of logs available in Azure:
|Log category||Log type||Usage||Integration|
|Activity logs||Control-plane events on Azure Resource Manager resources||Provides insight into the operations that were performed on resources in your subscription.||Rest API, Azure Monitor|
|Azure diagnostics logs||Frequent data about the operation of Azure Resource Manager resources in subscription||Provides insight into operations that your resource itself performed.||Azure Monitor, Stream|
|Azure AD reporting||Logs and reports||Reports user sign-in activities and system activity information about users and group management.||Graph API|
|Virtual machines and cloud services||Windows Event Log service and Linux Syslog||Captures system data and logging data on the virtual machines and transfers that data into a storage account of your choice.||Windows (using Windows Azure Diagnostics [WAD] storage) and Linux in Azure Monitor|
|Azure Storage Analytics||Storage logging, provides metrics data for a storage account||Provides insight into trace requests, analyzes usage trends, and diagnoses issues with your storage account.||REST API or the client library|
|Network Security Group (NSG) flow logs||JSON format, shows outbound and inbound flows on a per-rule basis||Displays information about ingress and egress IP traffic through a Network Security Group.||Azure Network Watcher|
|Application insight||Logs, exceptions, and custom diagnostics||Provides an application performance monitoring (APM) service for web developers on multiple platforms.||REST API, Power BI|
|Process data / security alerts||Azure Security Center alerts, Azure Log Analytics alerts||Provides security information and alerts.||REST APIs, JSON|
Azure activity logs provide insight into the operations that were performed on resources in your subscription. Activity logs were previously known as “audit logs” or “operational logs,” because they report control-plane events for your subscriptions.
Activity logs help you determine the “what, who, and when” for write operations (that is, PUT, POST, or DELETE). Activity logs also help you understand the status of the operation and other relevant properties. Activity logs do not include read (GET) operations.
In this article, PUT, POST, and DELETE refer to all the write operations that an activity log contains on the resources. For example, you can use the activity logs to find an error when you're troubleshooting issues or to monitor how a user in your organization modified a resource.
Integration scenarios for an activity log event:
Stream it to an event hub for ingestion by a third-party service or custom analytics solution such as PowerBI.
Analyze it in PowerBI by using the PowerBI content pack.
Save it to a storage account for archival or manual inspection. You can specify the retention time (in days) by using log profiles.
Query and view it in the Azure portal.
Query it via PowerShell cmdlet, Azure CLI, or REST API.
Export the activity log with log profiles to Log Analytics.
You can use a storage account or event hub namespace that is not in the same subscription as the one that's emitting the log. Whoever configures the setting must have the appropriate role-based access control (RBAC) access to both subscriptions.
Azure diagnostics logs
Azure diagnostics logs are emitted by a resource that provides rich, frequent data about the operation of that resource. The content of these logs varies by resource type. For example, Windows event system logs are a category of diagnostics logs for VMs, and blob, table, and queue logs are categories of diagnostics logs for storage accounts. Diagnostics logs differ from activity logs, which provide insight into the operations that were performed on resources in your subscription.
Azure diagnostics logs offer multiple configuration options, such as the Azure portal, PowerShell, Azure CLI, and the REST API.
Save them to a storage account for auditing or manual inspection. You can specify the retention time (in days) by using the diagnostics settings.
Analyze them with Log Analytics.
Supported services, schema for diagnostics logs and supported log categories per resource type
|Service||Schema and documentation||Resource type||Category|
|Azure Load Balancer||Log Analytics for Load Balancer (Preview)||Microsoft.Network/loadBalancers
|Network Security Groups||Log Analytics for Network Security Groups||Microsoft.Network/networksecuritygroups
|Azure Application Gateway||Diagnostics logging for Application Gateway||Microsoft.Network/applicationGateways
|Azure Key Vault||Key Vault logs||Microsoft.KeyVault/vaults||AuditEvent|
|Azure Search||Enabling and using Search Traffic Analytics||Microsoft.Search/searchServices||OperationLogs|
|Azure Data Lake Store||Access diagnostics logs for Data Lake Store||Microsoft.DataLakeStore/accounts
|Azure Data Lake Analytics||Access diagnostics logs for Data Lake Analytics||Microsoft.DataLakeAnalytics/accounts
|Azure Logic Apps||Logic Apps B2B custom tracking schema||Microsoft.Logic/workflows
|Azure Batch||Azure Batch diagnostics logs||Microsoft.Batch/batchAccounts||ServiceLog|
|Azure Automation||Log Analytics for Azure Automation||Microsoft.Automation/automationAccounts
|Azure Event Hubs||Event Hubs diagnostics logs||Microsoft.EventHub/namespaces
|Azure Stream Analytics||Job diagnostics logs||Microsoft.StreamAnalytics/streamingjobs
|Azure Service Bus||Service Bus diagnostics logs||Microsoft.ServiceBus/namespaces||OperationalLogs|
Azure Active Directory reporting
Azure Active Directory (Azure AD) includes security, activity, and audit reports for a user's directory. The Azure AD audit report helps you identify privileged actions that occurred in the user's Azure AD instance. Privileged actions include elevation changes (for example, role creation or password resets), changing policy configurations (for example, password policies), or changes to the directory configuration (for example, changes to domain federation settings).
The reports provide the audit record for the event name, the user who performed the action, the target resource affected by the change, and the date and time (in UTC). Users can retrieve the list of audit events for Azure AD via the Azure portal, as described in View your audit logs.
The included reports are listed in the following table:
|Security reports||Activity reports||Audit reports|
|Sign-ins from unknown sources||Application usage: summary||Directory audit report|
|Sign-ins after multiple failures||Application usage: detailed|
|Sign-ins from multiple geographies||Application dashboard|
|Sign-ins from IP addresses with suspicious activity||Account provisioning errors|
|Irregular sign-in activity||Individual user devices|
|Sign-ins from possibly infected devices||Individual user activity|
|Users with anomalous sign-in activity||Groups activity report|
|Password reset registration activity report|
|Password reset activity|
The data in these reports can be useful to your applications, such as Security Information and Event Management (SIEM) systems, audit, and business intelligence tools. The Azure AD reporting APIs provide programmatic access to the data through a set of REST-based APIs. You can call these APIs from various programming languages and tools.
Events in the Azure AD audit report are retained for 180 days.
For more information about report retention, see Azure AD report retention policies.
If you're interested in retaining your audit events longer, use the Reporting API to regularly pull audit events into a separate data store.
Virtual machine logs that use Azure Diagnostics
Azure Diagnostics is the capability within Azure that enables the collection of diagnostics data on a deployed application. You can use the diagnostics extension from any of several sources. Currently supported are Azure cloud service web and worker roles.
Azure virtual machines that are running Microsoft Windows and Service Fabric
You can enable Azure Diagnostics on a virtual machine by doing any of the following:
Azure Storage Analytics logs and provides metrics data for a storage account. You can use this data to trace requests, analyze usage trends, and diagnose issues with your storage account. Storage Analytics logging is available for the Azure Blob, Azure Queue, and Azure Table storage services. Storage Analytics logs detailed information about successful and failed requests to a storage service.
You can use this information to monitor individual requests and to diagnose issues with a storage service. Requests are logged on a best-effort basis. Log entries are created only if there are requests made against the service endpoint. For example, if a storage account has activity in its blob endpoint but not in its table or queue endpoints, only logs that pertain to the Blob storage service are created.
To use Storage Analytics, enable it individually for each service you want to monitor. You can enable it in the Azure portal. For more information, see Monitor a storage account in the Azure portal. You can also enable Storage Analytics programmatically via the REST API or the client library. Use the Set Service Properties operation to enable Storage Analytics individually for each service.
The aggregated data is stored in a well-known blob (for logging) and in well-known tables (for metrics), which you can access by using the Blob storage service and Table storage service APIs.
Storage Analytics has a 20-terabyte (TB) limit on the amount of stored data that is independent of the total limit for your storage account. All logs are stored in block blobs in a container named $logs, which is automatically created when you enable Storage Analytics for a storage account.
Storage Analytics logs the following types of authenticated and anonymous requests:
|Successful requests||Successful requests|
|Failed requests, including timeout, throttling, network, authorization, and other errors||Requests using a shared access signature, including failed and successful requests|
|Requests using a shared access signature, including failed and successful requests||Time-out errors for both client and server|
|Requests to analytics data||Failed GET requests with error code 304 (not modified)|
|Requests made by Storage Analytics itself, such as log creation or deletion, are not logged. A full list of the logged data is documented in Storage Analytics logged operations and status messages and Storage Analytics log format.||All other failed anonymous requests are not logged. A full list of the logged data is documented in Storage Analytics logged operations and status messages and Storage Analytics log format.|
Azure networking logs
Network logging and monitoring in Azure is comprehensive and covers two broad categories:
Network Watcher: Scenario-based network monitoring is provided with the features in Network Watcher. This service includes packet capture, next hop, IP flow verify, security group view, NSG flow logs. Scenario level monitoring provides an end to end view of network resources in contrast to individual network resource monitoring.
Resource monitoring: Resource level monitoring comprises four features, diagnostics logs, metrics, troubleshooting, and resource health. All these features are built at the network resource level.
Network Watcher is a regional service that enables you to monitor and diagnose conditions at a network scenario level in, to, and from Azure. Network diagnostics and visualization tools available with Network Watcher help you understand, diagnose, and gain insights to your network in Azure.
Network Security Group flow logging
NSG flow logs are a feature of Network Watcher that you can use to view information about ingress and egress IP traffic through an NSG. These flow logs are written in JSON format and show:
- Outbound and inbound flows on a per-rule basis.
- The NIC that the flow applies to.
- 5-tuple information about the flow: the source or destination IP, the source or destination port, and the protocol.
- Whether the traffic was allowed or denied.
Although flow logs target NSGs, they are not displayed in the same way as the other logs. Flow logs are stored only within a storage account.
The same retention policies that are seen on other logs apply to flow logs. Logs have a retention policy that you can set from 1 day to 365 days. If a retention policy is not set, the logs are maintained forever.
Periodic and spontaneous events are created by network resources and logged in storage accounts, and sent to an event hub or Log Analytics. The logs provide insights into the health of a resource. They can be viewed in tools such as Power BI and Log Analytics. To learn how to view diagnostics logs, see Log Analytics.
Network Watcher provides a diagnostics logs view. This view contains all networking resources that support diagnostics logging. From this view, you can enable and disable networking resources conveniently and quickly.
In addition to the previously mentioned logging capabilities, Network Watcher currently has the following capabilities:
Topology: Provides a network-level view that shows the various interconnections and associations between network resources in a resource group.
Variable packet capture: Captures packet data in and out of a virtual machine. Advanced filtering options and fine-tuning controls, such as time- and size-limitation settings, provide versatility. The packet data can be stored in a blob store or on the local disk in .cap file format.
IP flow verification: Checks to see whether a packet is allowed or denied based on flow information 5-tuple packet parameters (that is, destination IP, source IP, destination port, source port, and protocol). If the packet is denied by a security group, the rule and group that denied the packet is returned.
Next hop: Determines the next hop for packets being routed in the Azure network fabric, so that you can diagnose any misconfigured user-defined routes.
Security group view: Gets the effective and applied security rules that are applied on a VM.
Virtual network gateway and connection troubleshooting: Helps you troubleshoot virtual network gateways and connections.
Network subscription limits: Enables you to view network resource usage against limits.
Azure Application Insights is an extensible APM service for web developers on multiple platforms. Use it to monitor live web applications. It automatically detects performance anomalies. It includes powerful analytics tools to help you diagnose issues and to understand what users actually do with your app.
Application Insights is designed to help you continuously improve performance and usability.
It works for apps on a wide variety of platforms, including .NET, Node.js, and J2EE, whether they're hosted on-premises or in the cloud. It integrates with your DevOps process and has connection points with various development tools.
Application Insights is aimed at the development team, to help you understand how your app is performing and how it's being used. It monitors:
Request rates, response times, and failure rates: Find out which pages are most popular, at what times of day, and where your users are. See which pages perform best. If your response times and failure rates go high when there are more requests, you might have a resourcing problem.
Dependency rates, response times, and failure rates: Find out whether external services are slowing you down.
Exceptions: Analyze the aggregated statistics, or pick specific instances and drill into the stack trace and related requests. Both server and browser exceptions are reported.
Page views and load performance: Get reports from your users' browsers.
AJAX calls: Get webpage rates, response times, and failure rates.
User and session counts.
Performance counters: Get data from your Windows or Linux server machines, such as CPU, memory, and network usage.
Host diagnostics: Get data from Docker or Azure.
Diagnostics trace logs: Get data from your app, so that you can correlate trace events with requests.
Custom events and metrics: Get data that you write yourself in the client or server code, to track business events such as items sold or games won.
The following table lists and describes integration scenarios:
|Application map||The components of your app, with key metrics and alerts.|
|Diagnostics search for instance data||Search and filter events such as requests, exceptions, dependency calls, log traces, and page views.|
|Metrics Explorer for aggregated data||Explore, filter, and segment aggregated data such as rates of requests, failures, and exceptions; response times, page load times.|
|Dashboards||Mash up data from multiple resources and share with others. Great for multi-component applications, and for continuous display in the team room.|
|Live Metrics Stream||When you deploy a new build, watch these near-real-time performance indicators to make sure everything works as expected.|
|Analytics||Answer tough questions about your app's performance and usage by using this powerful query language.|
|Automatic and manual alerts||Automatic alerts adapt to your app's normal patterns of telemetry and are triggered when there's something outside the usual pattern. You can also set alerts on particular levels of custom or standard metrics.|
|Visual Studio||View performance data in the code. Go to code from stack traces.|
|Power BI||Integrate usage metrics with other business intelligence.|
|REST API||Write code to run queries over your metrics and raw data.|
|Continuous export||Bulk export of raw data to storage when it arrives.|
Azure Security Center alerts
Azure Security Center threat detection works by automatically collecting security information from your Azure resources, the network, and connected partner solutions. It analyzes this information, often correlating information from multiple sources, to identify threats. Security alerts are prioritized in Security Center along with recommendations on how to remediate the threat. For more information, see Azure Security Center.
Security Center employs advanced security analytics, which go far beyond signature-based approaches. It applies breakthroughs in large data and machine learning technologies to evaluate events across the entire cloud fabric. In this way, it detects threats that would be impossible to identify by using manual approaches and predicting the evolution of attacks. These security analytics include:
Integrated threat intelligence: Looks for known bad actors by applying global threat intelligence from Microsoft products and services, the Microsoft Digital Crimes Unit (DCU), the Microsoft Security Response Center (MSRC), and external feeds.
Behavioral analytics: Applies known patterns to discover malicious behavior.
Anomaly detection: Uses statistical profiling to build a historical baseline. It alerts on deviations from established baselines that conform to a potential attack vector.
Many security operations and incident response teams rely on a SIEM solution as the starting point for triaging and investigating security alerts. With Azure Log Integration, you can sync Security Center alerts and virtual machine security events, collected by Azure diagnostics and audit logs, with your Log Analytics or SIEM solution in near real time.
Log Analytics is a service in Azure that helps you collect and analyze data that's generated by resources in your cloud and on-premises environments. It gives you real-time insights by using integrated search and custom dashboards to readily analyze millions of records across all your workloads and servers, regardless of their physical location.
At the center of Log Analytics is the Log Analytics workspace, which is hosted in Azure. Log Analytics collects data in the workspace from connected sources by configuring data sources and adding solutions to your subscription. Data sources and solutions each create different record types, each with its own set of properties. But sources and solutions can still be analyzed together in queries to the workspace. This capability allows you to use the same tools and methods to work with a variety of data collected by a variety of sources.
Connected sources are the computers and other resources that generate the data that's collected by Log Analytics. Sources can include agents that are installed on Windows and Linux computers that connect directly, or agents in a connected System Center Operations Manager management group. Log Analytics can also collect data from an Azure storage account.
Data sources are the various kinds of data that's collected from each connected source. Sources include events and performance data from Windows and Linux agents, in addition to sources such as IIS logs and custom text logs. You configure each data source that you want to collect, and the configuration is automatically delivered to each connected source.
There are four ways to collect logs and metrics for Azure services:
Azure Diagnostics direct to Log Analytics (Diagnostics in the following table)
Azure Diagnostics to Azure storage to Log Analytics (Storage in the following table)
Connectors for Azure services (Connector in the following table)
Scripts to collect and then post data into Log Analytics (blank cells in the following table and for services that are not listed)
|Azure Application Gateway||Microsoft.Network/
|Diagnostics||Diagnostics||Azure Application Gateway Analytics|
|Application Insights||Connector||Connector||Application Insights Connector (Preview)|
|Azure Automation accounts||Microsoft.Automation/
|Azure Batch accounts||Microsoft.Batch/
|Classic cloud services||Storage||More information|
|Azure Data Lake Analytics||Microsoft.DataLakeAnalytics/
|Azure Data Lake Store||Microsoft.DataLakeStore/
|Azure Event Hub namespace||Microsoft.EventHub/
|Azure IoT Hub||Microsoft.Devices/
|Azure Key Vault||Microsoft.KeyVault/
|Diagnostics||Key Vault Analytics|
|Azure Load Balancer||Microsoft.Network/
|Azure Logic Apps||Microsoft.Logic/
|Network Security Groups||Microsoft.Network/
|Diagnostics||Azure Network Security Group analytics|
|Azure Recovery Services Analytics (Preview)|
|Service Bus namespace||Microsoft.ServiceBus/
|Diagnostics||Diagnostics||Service Bus Analytics (Preview)|
|Service Fabric||Storage||Service Fabric Analytics (Preview)|
|Storage||Script||Azure Storage Analytics (Preview)|
|Azure Virtual Machines||Microsoft.Compute/
|Virtual machine scale sets||Microsoft.Compute/
|Web server farms||Microsoft.Web/
Log Integration with on-premises SIEM systems
With Azure Log Integration, you can integrate raw logs from your Azure resources with your on-premises SIEM system.
Log Integration collects Azure diagnostics from your Windows virtual machines, Azure activity logs, Azure Security Center alerts, and Azure resource provider logs. This integration provides a unified dashboard for all your assets, whether they're on-premises or in the cloud, so that you can aggregate, correlate, analyze, and alert for security events.
Log Integration currently supports the integration of Azure activity logs, Windows event logs from Windows virtual machines with your Azure subscription, Azure Security Center alerts, Azure diagnostics logs, and Azure AD audit logs.
|Log type||Log Analytics supporting JSON (Splunk, ArcSight, and IBM QRadar)|
|Azure AD audit logs||Yes|
|Security Center alerts||Yes|
|Diagnostics logs (resource logs)||Yes|
|VM logs||Yes, via forwarded events and not through JSON|
Get started with Azure Log Integration: This tutorial walks you through installing Azure Log Integration and integrating logs from Azure storage, Azure activity logs, Azure Security Center alerts, and Azure AD audit logs.
Integration scenarios for SIEM:
Partner configuration steps: This blog post shows you how to configure Azure Log Integration to work with partner solutions Splunk, HP ArcSight, and IBM QRadar.
Azure Log Integration FAQ: This article answers questions about Azure Log Integration.
Integrating Security Center alerts with Azure Log Integration: This article discusses how to sync Security Center alerts, virtual machine security events collected by Azure diagnostics logs, and Azure audit logs with your Log Analytics or SIEM solution.
Auditing and logging: Protect data by maintaining visibility and responding quickly to timely security alerts.
Security logging and audit-log collection within Azure: Enforce these settings to ensure that your Azure instances are collecting the correct security and audit logs.
Configure audit settings for a site collection: If you're a site collection administrator, retrieve the history of individual users' actions and the history of actions taken during a particular date range.
Search the audit log in the Office 365 Security & Compliance Center: Use the Office 365 Security & Compliance Center to search the unified audit log and view user and administrator activity in your Office 365 organization.