Troubleshoot changes in your environment

In this tutorial, you learn how to troubleshoot changes on an Azure virtual machine. By enabling Change tracking, you can track changes to software, files, Linux daemons, Windows Services, and Windows Registry keys on your computers. Identifying these configuration changes can help you pinpoint operational issues across your environment.

In this tutorial you learn how to:

  • Onboard a VM for Change tracking and Inventory
  • Search change logs for stopped services
  • Configure change tracking
  • Enable Activity log connection
  • Trigger an event
  • View changes
  • Configure alerts

Prerequisites

To complete this tutorial, you need:

Sign in to Azure

Sign in to the Azure portal at https://portal.azure.com.

Enable Change tracking and Inventory

First you need to enable Change tracking and Inventory for your VM for this tutorial. If you have previously enabled another automation solution for a VM, this step is not necessary.

  1. On the left menu, select Virtual machines and select a VM from the list
  2. On the left menu, under the OPERATIONS section, click Inventory. The Change tracking page opens.

Enable change The Change Tracking screen opens. Configure the location, Log analytics workspace, and Automation account to use and click Enable. If the fields are grayed out, that means another automation solution is enabled for the VM and the same workspace and Automation account must be used.

A Log Analytics workspace is used to collect data that is generated by features and services such as Inventory. The workspace provides a single location to review and analyze data from multiple sources.

During onboarding, the VM is provisioned with the Microsoft Monitoring Agent (MMA) and hybrid worker. This agent is used to communicate with the VM and obtain information about installed software.

Enabling the solution can take up to 15 minutes. During this time, you shouldn't close the browser window. After the solution is enabled, information about installed software and changes on the VM flows to Log Analytics. It can take between 30 minutes and 6 hours for the data to be available for analysis.

Using Change tracking in Log Analytics

Change tracking generates log data that is sent to Log Analytics. To search the logs by running queries, select Log Analytics at the top of the Change tracking window. Change tracking data is stored under the type ConfigurationChange. The following sample Log Analytics query returns all the Windows Services that have been stopped.

ConfigurationChange
| where ConfigChangeType == "WindowsServices" and SvcState == "Stopped"

To learn more about running and searching log files in Log Analytics, see Azure Log Analytics.

Configure Change tracking

Change tracking gives you the ability to track configuration changes on your VM. The following steps show you how to configure tracking of registry keys and files.

To choose which files and Registry keys to collect and track, select Edit settings at the top of the Change tracking page.

Note

Inventory and Change tracking use the same collection settings, and settings are configured on a workspace level.

In the Workspace Configuration window, add the Windows Registry keys, Windows files, or Linux files to be tracked, as outlined in the next three sections.

Add a Windows Registry key

  1. On the Windows Registry tab, select Add. The Add Windows Registry for Change Tracking window opens.

  2. On the Add Windows Registry for Change Tracking, enter the information for the key to track and click Save

Property Description
Enabled Determines if the setting is applied
Item Name Friendly name of the file to be tracked
Group A group name for logically grouping files
Windows Registry Key The path to check for the file For example: "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\User Shell Folders\Common Startup"

Add a Windows file

  1. On the Windows Files tab, select Add. The Add Windows File for Change Tracking window opens.

  2. On the Add Windows File for Change Tracking, enter the information for the file or directory to track and click Save

Property Description
Enabled Determines if the setting is applied
Item Name Friendly name of the file to be tracked
Group A group name for logically grouping files
Enter Path The path to check for the file For example: "c:\temp\*.txt"
You can also use environment variables such as "%winDir%\System32\*.*"
Recursion Determines if recursion is used when looking for the item to be tracked.
Upload file content for all settings Turns on or off file content upload on tracked changes. Available options: True or False.

Add a Linux file

  1. On the Linux Files tab, select Add. The Add Linux File for Change Tracking window opens.

  2. On the Add Linux File for Change Tracking, enter the information for the file or directory to track and click Save

Property Description
Enabled Determines if the setting is applied
Item Name Friendly name of the file to be tracked
Group A group name for logically grouping files
Enter Path The path to check for the file For example: "/etc/*.conf"
Path Type Type of item to be tracked, possible values are File and Directory
Recursion Determines if recursion is used when looking for the item to be tracked.
Use Sudo This setting determines if sudo is used when checking for the item.
Links This setting determines how symbolic links dealt with when traversing directories.
Ignore - Ignores symbolic links and does not include the files/directories referenced
Follow - Follows the symbolic links during recursion and also includes the files/directories referenced
Manage - Follows the symbolic links and allows alter the treatment of returned content
Upload file content for all settings Turns on or off file content upload on tracked changes. Available options: True or False.

Note

The "Manage" links option is not recommended. File content retrieval is not supported.

Enable Activity log connection

From the Change tracking page on your VM, select Manage Activity Log Connection. This task opens the Azure Activity log page. Select Connect to connect Change tracking to the Azure activity log for your VM.

With this setting enabled, navigate to the Overview page for your VM and select Stop to stop your VM. When prompted, select Yes to stop the VM. When it is deallocated, select Start to restart your VM.

Stopping and starting a VM logs an event in its activity log. Navigate back to the Change tracking page. Select the Events tab at the bottom of the page. After a while, the events shown in the chart and the table. Like in the preceding step, each event can be selected to view detailed information on the event.

Viewing change details in the portal

View changes

Once the Change tracking and Inventory solution is enabled, you can view the results on the Change tracking page.

From within your VM, select Change tracking under OPERATIONS.

Screenshot that shows the list of changes to the VM

The chart shows changes that have occurred over time. After you have added an Activity Log connection, the line graph at the top displays Azure Activity Log events. Each row of bar graphs represents a different trackable Change type. These types are Linux daemons, files, Windows Registry keys, software, and Windows services. The change tab shows the details for the changes shown in the visualization in descending order of time that the change occurred (most recent first). The Events tab, the table displays the connected Activity Log events and their corresponding details with the most recent first.

You can see in the results, that there were multiple changes to the system, including changes to services and software. You can use the filters at the top of the page to filter the results by Change type or by a time range.

Select a WindowsServices change, this opens the Change Details window. The change details window shows details about the change and the values before and after the change. In this instance, the Software Protection service was stopped.

Viewing change details in the portal

Configure alerts

Viewing changes in the Azure portal can be helpful, but being able to be alerted when a change occurs, such as a stopped service is more beneficial.

To add an alert for a stopped service, in the Azure portal, go to Monitor. And then under Shared Services, select Alerts and click + New alert rule

Click Select to choose a resource. On the Select a resource page, select Log Analytics from the Filter by resource type drop-down. Select your Log Analytics workspace, and then select Done.

Select a resource

Click Add condition, on the Configure signal logic page, in the table, select Custom log search. Enter the following query in the Search query text box:

ConfigurationChange | where ConfigChangeType == "WindowsServices" and SvcName == "W3SVC" and SvcState == "Stopped" | summarize by Computer

This query returns the computers that had the W3SVC service stopped in the specified timeframe.

Under Alert logic, for Threshold, enter 0. When you're finished, select Done.

Configure signal logic

Under Action Groups, select Create New. An action group is a group of actions that you can use across multiple alerts. The actions can include but are not limited to email notifications, runbooks, webhooks, and many more. To learn more about action groups, see Create and manage action groups.

Under Alert details, enter a name and description for the alert. Set Severity to Informational(Sev 2), Warning(Sev 1), or Critical(Sev 0).

In the Action group name box, enter a name for the alert and a short name. The short name is used in place of a full action group name when notifications are sent by using this group.

Under Actions, enter a name for the action, like Email Administrators. Under ACTION TYPE, select Email/SMS/Push/Voice. Under DETAILS, select Edit details.

Add action group

In the Email/SMS/Push/Voice pane, enter a name. Select the Email check box, and then enter a valid email address. Click OK on the Email/SMS/Push/Voice page, and then click OK on the Add action group page.

To customize the subject of the alert email, under Create rule, under Customize Actions, select Email subject. When you're finished, select Create alert rule. The alert tells you when an update deployment succeeds, and which machines were part of that update deployment run.

The following image is an example email received wen the W3SVC service stops.

email

Next steps

In this tutorial you learned how to:

  • Onboard a VM for Change tracking and Inventory
  • Search change logs for stopped services
  • Configure change tracking
  • Enable Activity log connection
  • Trigger an event
  • View changes
  • Configure alerts

Continue to the overview for the Change tracking and Inventory solution to learn more about it.