Azure Log Integration with Azure Diagnostics logging and Windows event forwarding

Azure Log Integration provides customers with an alternative if an Azure Monitor connector isn't available from their Security Incident and Event Management (SIEM) vendor. Azure Log Integration makes Azure logs available to your SIEM so you can create a unified security dashboard for all your assets.

Note

For more information about Azure Monitor, see Get started with Azure Monitor. For more information about the status of an Azure Monitor connector, contact your SIEM vendor.

Important

If your primary interest is collecting virtual machine logs, most SIEM vendors include this option in their solution. Using the SIEM vendor's connector is always the preferred alternative.

This article helps you get started with Azure Log Integration. It focuses on installing the Azure Log Integration service and integrating the service with Azure Diagnostics. The Azure Log Integration service then collects Windows Event Log information from the Windows Security Event channel from virtual machines deployed in an Azure infrastructure as a service. This is similar to event forwarding that you might use in an on-premises system.

Note

Integrating the output of Azure Log Integration with an SIEM is done by the SIEM itself. For more information, see Integrate Azure Log Integration with your on-premises SIEM.

The Azure Log Integration service runs on either a physical or a virtual computer running Windows Server 2008 R2 or later (Windows Server 2016 or Windows Server 2012 R2 is preferred).

A physical computer can run on-premises or on a hosting site. If you choose to run the Azure Log Integration service on a virtual machine, the virtual machine can be located on-premises or in a public cloud, such as in Microsoft Azure.

The physical or virtual machine running the Azure Log Integration service requires network connectivity to the Azure public cloud. This article provides details about the required configuration.

Prerequisites

At a minimum, installing Azure Log Integration requires the following items:

  • An Azure subscription. If you don't have one, you can sign up for a free account.
  • A storage account that can be used for Windows Azure Diagnostics (WAD) logging. You can use a preconfigured storage account or create a new storage account. Later in this article, we describe how to configure the storage account.

    Note

    Depending on your scenario, a storage account might not be required. For the Azure Diagnostics scenario covered in this article, a storage account is required.

  • Two systems:

    • A machine that runs the Azure Log Integration service. This machine collects all the log information that later is imported into your SIEM. This system:
      • Can be on-premises or hosted in Microsoft Azure.
      • Must be running an x64 version of Windows Server 2008 R2 SP1 or later, and have Microsoft .NET 4.5.1 installed. To determine the .NET version installed, see Determine which .NET Framework versions are installed.
      • Must have connectivity to the Azure Storage account that's used for Azure Diagnostics logging. Later in this article, we describe how to confirm connectivity.
    • A machine that you want to monitor. This is a VM running as an Azure virtual machine. The logging information from this machine is sent to the Azure Log Integration service machine.

For a quick demonstration of how to create a virtual machine by using the Azure portal, take a look at the following video:

Deployment considerations

During testing, you can use any system that meets the minimum operating system requirements. For a production environment, the load might require you to plan for scaling up or scaling out.

You can run multiple instances of the Azure Log Integration service. However, you can run only one instance of the service per physical or virtual machine. In addition, you can load-balance Azure Diagnostics storage accounts for WAD. The number of subscriptions to provide to the instances is based on your capacity.

Note

Currently, we don't have specific recommendations about when to scale out instances of Azure Log Integration machines (that is, machines running the Azure Log Integration service), or for storage accounts or subscriptions. Make scaling decisions based on your performance observations in each of these areas.

To help improve performance, you also have the option to scale up the Azure Log Integration service. The following performance metrics can help you size the machines that you choose to run the Azure Log Integration service:

  • On an 8-processor (core) machine, a single instance of Azure Log Integration can process about 24 million events per day (approximately 1 million events per hour).
  • On a 4-processor (core) machine, a single instance of Azure Log Integration can process about 1.5 million events per day (approximately 62,500 events per hour).

Install Azure Log Integration

To install Azure Log Integration, download the Azure Log Integration installation file. Complete the setup process. Choose whether to provide telemetry information to Microsoft.

The Azure Log Integration service collects telemetry data from the machine on which it's installed.

Telemetry data that's collected includes the following:

  • Exceptions that occur during execution of Azure Log Integration.
  • Metrics about the number of queries and events processed.
  • Statistics about which Azlog.exe command-line options are used.

Note

We recommend that you allow Microsoft to collect telemetry data. You can turn off the collection of telemetry data by clearing the Allow Microsoft to collect telemetry data check box.

Screenshot of the installation pane, with the telemetry check box selected

The installation process is covered in the following video:

Post-installation and validation steps

After you complete basic setup, you're ready to perform post-installation and validation steps:

  1. Open PowerShell as an administrator. Then, go to C:\Program Files\Microsoft Azure Log Integration.
  2. Import the Azure Log Integration cmdlets. To import the cmdlets, run the script LoadAzlogModule.ps1. Enter .\LoadAzlogModule.ps1, and then press Enter (note the use of .\ in this command). You should see something like what appears in the following figure:

    Screenshot of the output of the LoadAzlogModule.ps1 command

  3. Next, configure Azure Log Integration to use a specific Azure environment. An Azure environment is the type of Azure cloud datacenter that you want to work with. Although there are several Azure environments, currently, the relevant options are either AzureCloud or AzureUSGovernment. Running PowerShell as an administrator, make sure that you are in C:\Program Files\Microsoft Azure Log Integration. Then, run this command:

    Set-AzlogAzureEnvironment -Name AzureCloud (for AzureCloud)

    If you want to use the US Government Azure cloud, use AzureUSGovernment for the -Name variable. Currently, other Azure clouds aren't supported.

    Note

    You don't receive feedback when the command succeeds.

  4. Before you can monitor a system, you need the name of the storage account that's used for Azure Diagnostics. In the Azure portal, go to Virtual machines. Look for the virtual machine that you monitor. In the Properties section, select Diagnostic Settings. Then, select Agent. Make note of the storage account name that's specified. You need this account name for a later step.

    Screenshot of the Azure Diagnostics Settings pane

    Screenshot of Enable guest-level monitoring button

    Note

    If monitoring wasn't enabled when the virtual machine was created, you can enable it as shown in the preceding image.

  5. Now, go back to the Azure Log Integration machine. Verify that you have connectivity to the storage account from the system where you installed Azure Log Integration. The computer running the Azure Log Integration service needs access to the storage account to retrieve information that's logged by Azure Diagnostics on each of the monitored systems. To verify connectivity:

    1. Download Azure Storage Explorer.
    2. Complete setup.
    3. When installation is finished, select Next. Leave the Launch Microsoft Azure Storage Explorer check box selected.
    4. Sign in to Azure.
    5. Verify that you can see the storage account that you configured for Azure Diagnostics:

      Screenshot of storage accounts in Storage Explorer

    6. A few options appear under storage accounts. Under Tables, you should see a table called WADWindowsEventLogsTable.

    If monitoring wasn't enabled when the virtual machine was created, you can enable it, as described earlier.

Integrate Azure Diagnostics logging

In this step, you configure the machine running the Azure Log Integration service to connect to the storage account that contains the log files.

To complete this step, you need a few things:

  • FriendlyNameForSource: A friendly name that you can apply to the storage account that you've configured for the virtual machine to store information from Azure Diagnostics.
  • StorageAccountName: The name of the storage account that you specified when you configured Azure Diagnostics.
  • StorageKey: The storage key for the storage account where the Azure Diagnostics information is stored for this virtual machine.

To obtain the storage key, complete the following steps:

  1. Go to the Azure portal.
  2. In the navigation pane, select All services.
  3. In the Filter box, enter Storage. Then, select Storage accounts.

    Screenshot that shows storage accounts in All services

  4. A list of storage accounts appears. Double-click the account that you assigned to log storage.

    Screenshot that shows a list of storage accounts

  5. Under Settings, select Access keys.

    Screenshot that shows the Access keys option in the menu

  6. Copy key1, and then save it in a secure location that you can access for the following step.

  7. On the server where you installed Azure Log Integration, open a Command Prompt window as an administrator. (Be sure to open a Command Prompt window as an administrator, and not PowerShell).
  8. Go to C:\Program Files\Microsoft Azure Log Integration.
  9. Run this command: Azlog source add <FriendlyNameForTheSource> WAD <StorageAccountName> <StorageKey>.

    Example:

    Azlog source add Azlogtest WAD Azlog9414 fxxxFxxxxxxxxywoEJK2xxxxxxxxxixxxJ+xVJx6m/X5SQDYc4Wpjpli9S9Mm+vXS2RVYtp1mes0t9H5cuqXEw==

    If you want the subscription ID to show up in the event XML, append the subscription ID to the friendly name:

    Azlog source add <FriendlyNameForTheSource>.<SubscriptionID> WAD <StorageAccountName> <StorageKey>

    Example:

    Azlog source add Azlogtest.YourSubscriptionID WAD Azlog9414 fxxxFxxxxxxxxywoEJK2xxxxxxxxxixxxJ+xVJx6m/X5SQDYc4Wpjpli9S9Mm+vXS2RVYtp1mes0t9H5cuqXEw==

Note

Wait up to 60 minutes, and then view the events that are pulled from the storage account. To view the events, in Azure Log Integration, select Event Viewer > Windows Logs > Forwarded Events.

The following video covers the preceding steps:

If data isn't showing up in the Forwarded Events folder

If data isn't showing up in the Forwarded Events folder after an hour, complete these steps:

  1. Check the machine that's running the Azure Log Integration service. Confirm that it can access Azure. To test connectivity, in a browser, try to go to the Azure portal.
  2. Make sure that the user account Azlog has write permission for the folder users\Azlog.
    1. Open File Explorer.
    2. Go to C:\users.
    3. Right-click C:\users\Azlog.
    4. Select Security.
    5. Select NT Service\Azlog. Check the permissions for the account. If the account is missing from this tab, or if the appropriate permissions aren't showing, you can grant the account permissions on this tab.
  3. When you run the command Azlog source list, make sure that the storage account that was added in the command Azlog source add is listed in the output.
  4. To see if any errors are reported from the Azure Log Integration service, go to Event Viewer > Windows Logs > Application.

If you run into any issues during installation and configuration, you can create a support request. For the service, select Log Integration.

Another support option is the Azure Log Integration MSDN forum. In the MSDN forum, the community can provide support by answering questions and sharing tips and tricks about how to get the most out of Azure Log Integration. The Azure Log Integration team also monitors this forum. They help whenever they can.

Next steps

To learn more about Azure Log Integration, see the following articles: