Use packet capture for proactive network monitoring with alerts and Azure Functions

Network Watcher packet capture creates capture sessions to track traffic in and out of virtual machines. The capture file can have a filter that is defined to track only the traffic that you want to monitor. This data is then stored in a storage blob or locally on the guest machine.

This capability can be started remotely from other automation scenarios such as Azure Functions. Packet capture gives you the capability to run proactive captures based on defined network anomalies. Other uses include gathering network statistics, getting information about network intrusions, debugging client-server communications, and more.

Resources that are deployed in Azure run 24/7. You and your staff cannot actively monitor the status of all resources 24/7. For example, what happens if an issue occurs at 2 AM?

By using Network Watcher, alerting, and functions from within the Azure ecosystem, you can proactively respond with the data and tools to solve problems in your network.

Scenario

Prerequisites

Scenario

In this example, your VM is sending more TCP segments than usual, and you want to be alerted. TCP segments are used as an example here, but you can use any alert condition.

When you are alerted, you want to receive packet-level data to understand why communication has increased. Then you can take steps to return the virtual machine to regular communication.

This scenario assumes that you have an existing instance of Network Watcher and a resource group with a valid virtual machine.

The following list is an overview of the workflow that takes place:

  1. An alert is triggered on your VM.
  2. The alert calls your Azure function via a webhook.
  3. Your Azure function processes the alert and starts a Network Watcher packet capture session.
  4. The packet capture runs on the VM and collects traffic.
  5. The packet capture file is uploaded to a storage account for review and diagnosis.

To automate this process, we create and connect an alert on our VM to trigger when the incident occurs. We also create a function to call into Network Watcher.

This scenario does the following:

  • Creates an Azure function that starts a packet capture.
  • Creates an alert rule on a virtual machine and configures the alert rule to call the Azure function.

Create an Azure function

The first step is to create an Azure function to process the alert and create a packet capture.

  1. In the Azure portal, select Create a resource > Compute > Function App.

    Creating a function app

  2. On the Function App blade, enter the following values, and then select OK to create the app:

    Setting Value Details
    App name PacketCaptureExample The name of the function app.
    Subscription [Your subscription]The subscription for which to create the function app.
    Resource Group PacketCaptureRG The resource group to contain the function app.
    Hosting Plan Consumption Plan The type of plan your function app uses. Options are Consumption or Azure App Service plan.
    Location Central US The region in which to create the function app.
    Storage Account {autogenerated} The storage account that Azure Functions needs for general-purpose storage.
  3. On the PacketCaptureExample Function Apps blade, select Functions > Custom function >+.

  4. Select HttpTrigger-Powershell, and then enter the remaining information. Finally, to create the function, select Create.

    Setting Value Details
    Scenario Experimental Type of scenario
    Name your function AlertPacketCapturePowerShell Name of the function
    Authorization level Function Authorization level for the function

Functions example

Note

The PowerShell template is experimental and does not have full support.

Customizations are required for this example and are explained in the following steps.

Add modules

To use Network Watcher PowerShell cmdlets, upload the latest PowerShell module to the function app.

  1. On your local machine with the latest Azure PowerShell modules installed, run the following PowerShell command:

    (Get-Module AzureRM.Network).Path
    

    This example gives you the local path of your Azure PowerShell modules. These folders are used in a later step. The modules that are used in this scenario are:

    • AzureRM.Network

    • AzureRM.Profile

    • AzureRM.Resources

      PowerShell folders

  2. Select Function app settings > Go to App Service Editor.

    Function app settings

  3. Right-click the AlertPacketCapturePowershell folder, and then create a folder called azuremodules.

  4. Create a subfolder for each module that you need.

    Folder and subfolders

    • AzureRM.Network

    • AzureRM.Profile

    • AzureRM.Resources

  5. Right-click the AzureRM.Network subfolder, and then select Upload Files.

  6. Go to your Azure modules. In the local AzureRM.Network folder, select all the files in the folder. Then select OK.

  7. Repeat these steps for AzureRM.Profile and AzureRM.Resources.

    Upload files

  8. After you've finished, each folder should have the PowerShell module files from your local machine.

    PowerShell files

Authentication

To use the PowerShell cmdlets, you must authenticate. You configure authentication in the function app. To configure authentication, you must configure environment variables and upload an encrypted key file to the function app.

Note

This scenario provides just one example of how to implement authentication with Azure Functions. There are other ways to do this.

Encrypted credentials

The following PowerShell script creates a key file called PassEncryptKey.key. It also provides an encrypted version of the password that's supplied. This password is the same password that is defined for the Azure Active Directory application that's used for authentication.

#Variables
$keypath = "C:\temp\PassEncryptKey.key"
$AESKey = New-Object Byte[] 32
$Password = "<insert a password here>"

#Keys
[Security.Cryptography.RNGCryptoServiceProvider]::Create().GetBytes($AESKey) 
Set-Content $keypath $AESKey

#Get encrypted password
$secPw = ConvertTo-SecureString -AsPlainText $Password -Force
$AESKey = Get-content $KeyPath
$Encryptedpassword = $secPw | ConvertFrom-SecureString -Key $AESKey
$Encryptedpassword

In the App Service Editor of the function app, create a folder called keys under AlertPacketCapturePowerShell. Then upload the PassEncryptKey.key file that you created in the previous PowerShell sample.

Functions key

Retrieve values for environment variables

The final requirement is to set up the environment variables that are necessary to access the values for authentication. The following list shows the environment variables that are created:

  • AzureClientID

  • AzureTenant

  • AzureCredPassword

AzureClientID

The client ID is the Application ID of an application in Azure Active Directory.

  1. If you don't already have an application to use, run the following example to create an application.

    $app = New-AzureRmADApplication -DisplayName "ExampleAutomationAccount_MF" -HomePage "https://exampleapp.com" -IdentifierUris "https://exampleapp1.com/ExampleFunctionsAccount" -Password "<same password as defined earlier>"
    New-AzureRmADServicePrincipal -ApplicationId $app.ApplicationId
    Start-Sleep 15
    New-AzureRmRoleAssignment -RoleDefinitionName Contributor -ServicePrincipalName $app.ApplicationId
    

    Note

    The password that you use when creating the application should be the same password that you created earlier when saving the key file.

  2. In the Azure portal, select Subscriptions. Select the subscription to use, and then select Access control (IAM).

    Functions IAM

  3. Choose the account to use, and then select Properties. Copy the Application ID.

    Functions Application ID

AzureTenant

Obtain the tenant ID by running the following PowerShell sample:

(Get-AzureRmSubscription -SubscriptionName "<subscriptionName>").TenantId

AzureCredPassword

The value of the AzureCredPassword environment variable is the value that you get from running the following PowerShell sample. This example is the same one that's shown in the preceding Encrypted credentials section. The value that's needed is the output of the $Encryptedpassword variable. This is the service principal password that you encrypted by using the PowerShell script.

#Variables
$keypath = "C:\temp\PassEncryptKey.key"
$AESKey = New-Object Byte[] 32
$Password = "<insert a password here>"

#Keys
[Security.Cryptography.RNGCryptoServiceProvider]::Create().GetBytes($AESKey) 
Set-Content $keypath $AESKey

#Get encrypted password
$secPw = ConvertTo-SecureString -AsPlainText $Password -Force
$AESKey = Get-content $KeyPath
$Encryptedpassword = $secPw | ConvertFrom-SecureString -Key $AESKey
$Encryptedpassword

Store the environment variables

  1. Go to the function app. Then select Function app settings > Configure app settings.

    Configure app settings

  2. Add the environment variables and their values to the app settings, and then select Save.

    App settings

Add PowerShell to the function

It's now time to make calls into Network Watcher from within the Azure function. Depending on the requirements, the implementation of this function can vary. However, the general flow of the code is as follows:

  1. Process input parameters.
  2. Query existing packet captures to verify limits and resolve name conflicts.
  3. Create a packet capture with appropriate parameters.
  4. Poll packet capture periodically until it's complete.
  5. Notify the user that the packet capture session is complete.

The following example is PowerShell code that can be used in the function. There are values that need to be replaced for subscriptionId, resourceGroupName, and storageAccountName.

            #Import Azure PowerShell modules required to make calls to Network Watcher
            Import-Module "D:\home\site\wwwroot\AlertPacketCapturePowerShell\azuremodules\AzureRM.Profile\AzureRM.Profile.psd1" -Global
            Import-Module "D:\home\site\wwwroot\AlertPacketCapturePowerShell\azuremodules\AzureRM.Network\AzureRM.Network.psd1" -Global
            Import-Module "D:\home\site\wwwroot\AlertPacketCapturePowerShell\azuremodules\AzureRM.Resources\AzureRM.Resources.psd1" -Global

            #Process alert request body
            $requestBody = Get-Content $req -Raw | ConvertFrom-Json

            #Storage account ID to save captures in
            $storageaccountid = "/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.Storage/storageAccounts/{storageAccountName}"

            #Packet capture vars
            $packetcapturename = "PSAzureFunction"
            $packetCaptureLimit = 10
            $packetCaptureDuration = 10

            #Credentials
            $tenant = $env:AzureTenant
            $pw = $env:AzureCredPassword
            $clientid = $env:AzureClientId
            $keypath = "D:\home\site\wwwroot\AlertPacketCapturePowerShell\keys\PassEncryptKey.key"

            #Authentication
            $secpassword = $pw | ConvertTo-SecureString -Key (Get-Content $keypath)
            $credential = New-Object System.Management.Automation.PSCredential ($clientid, $secpassword)
            Connect-AzureRmAccount -ServicePrincipal -Tenant $tenant -Credential $credential #-WarningAction SilentlyContinue | out-null


            #Get the VM that fired the alert
            if($requestBody.context.resourceType -eq "Microsoft.Compute/virtualMachines")
            {
                Write-Output ("Subscription ID: {0}" -f $requestBody.context.subscriptionId)
                Write-Output ("Resource Group:  {0}" -f $requestBody.context.resourceGroupName)
                Write-Output ("Resource Name:  {0}" -f $requestBody.context.resourceName)
                Write-Output ("Resource Type:  {0}" -f $requestBody.context.resourceType)

                #Get the Network Watcher in the VM's region
                $nw = Get-AzurermResource | Where {$_.ResourceType -eq "Microsoft.Network/networkWatchers" -and $_.Location -eq $requestBody.context.resourceRegion}
                $networkWatcher = Get-AzureRmNetworkWatcher -Name $nw.Name -ResourceGroupName $nw.ResourceGroupName

                #Get existing packetCaptures
                $packetCaptures = Get-AzureRmNetworkWatcherPacketCapture -NetworkWatcher $networkWatcher

                #Remove existing packet capture created by the function (if it exists)
                $packetCaptures | %{if($_.Name -eq $packetCaptureName)
                { 
                    Remove-AzureRmNetworkWatcherPacketCapture -NetworkWatcher $networkWatcher -PacketCaptureName $packetCaptureName
                }}

                #Initiate packet capture on the VM that fired the alert
                if ((Get-AzureRmNetworkWatcherPacketCapture -NetworkWatcher $networkWatcher).Count -lt $packetCaptureLimit){
                    echo "Initiating Packet Capture"
                    New-AzureRmNetworkWatcherPacketCapture -NetworkWatcher $networkWatcher -TargetVirtualMachineId $requestBody.context.resourceId -PacketCaptureName $packetCaptureName -StorageAccountId $storageaccountid -TimeLimitInSeconds $packetCaptureDuration
                    Out-File -Encoding Ascii -FilePath $res -inputObject "Packet Capture created on ${requestBody.context.resourceID}"
                }
            } 

Retrieve the function URL

  1. After you've created your function, configure your alert to call the URL that's associated with the function. To get this value, copy the function URL from your function app.

    Finding the function URL

  2. Copy the function URL for your function app.

    Copying the function URL

If you require custom properties in the payload of the webhook POST request, refer to Configure a webhook on an Azure metric alert.

Configure an alert on a VM

Alerts can be configured to notify individuals when a specific metric crosses a threshold that's assigned to it. In this example, the alert is on the TCP segments that are sent, but the alert can be triggered for many other metrics. In this example, an alert is configured to call a webhook to call the function.

Create the alert rule

Go to an existing virtual machine, and then add an alert rule. More detailed documentation about configuring alerts can be found at Create alerts in Azure Monitor for Azure services - Azure portal. Enter the following values in the Alert rule blade, and then select OK.

Setting Value Details
Name TCP_Segments_Sent_Exceeded Name of the alert rule.
Description TCP segments sent exceeded threshold The description for the alert rule.
Metric TCP segments sent The metric to use to trigger the alert.
Condition Greater than The condition to use when evaluating the metric.
Threshold 100 The value of the metric that triggers the alert. This value should be set to a valid value for your environment.
Period Over the last five minutes Determines the period in which to look for the threshold on the metric.
Webhook [webhook URL from function app] The webhook URL from the function app that was created in the previous steps.

Note

The TCP segments metric is not enabled by default. Learn more about how to enable additional metrics by visiting Enable monitoring and diagnostics.

Review the results

After the criteria for the alert triggers, a packet capture is created. Go to Network Watcher, and then select Packet capture. On this page, you can select the packet capture file link to download the packet capture.

View packet capture

If the capture file is stored locally, you can retrieve it by signing in to the virtual machine.

For instructions about downloading files from Azure storage accounts, see Get started with Azure Blob storage using .NET. Another tool you can use is Storage Explorer.

After your capture has been downloaded, you can view it by using any tool that can read a .cap file. Following are links to two of these tools:

Next steps

Learn how to view your packet captures by visiting Packet capture analysis with Wireshark.