Collect custom metrics for a Linux VM with the InfluxData Telegraf agent
By using Azure Monitor, you can collect custom metrics via your application telemetry, an agent running on your Azure resources, or even outside-in monitoring systems. Then you can submit them directly to Azure Monitor. This article provides instructions on how to deploy the InfluxData Telegraf agent on a Linux VM in Azure and configure the agent to publish metrics to Azure Monitor.
InfluxData Telegraf agent
Telegraf is a plug-in-driven agent that enables the collection of metrics from over 150 different sources. Depending on what workloads run on your VM, you can configure the agent to leverage specialized input plug-ins to collect metrics. Examples are MySQL, NGINX, and Apache. By using output plug-ins, the agent can then write to destinations that you choose. The Telegraf agent has integrated directly with the Azure Monitor custom metrics REST API. It supports an Azure Monitor output plug-in. By using this plug-in, the agent can collect workload-specific metrics on your Linux VM and submit them as custom metrics to Azure Monitor.
Custom Metrics are not supported in all regions. Supported regions are listed here
Send custom metrics
For this tutorial, we deploy a Linux VM that runs the Ubuntu 16.04 LTS operating system. The Telegraf agent is supported for most Linux operating systems. Both Debian and RPM packages are available along with unpackaged Linux binaries on the InfluxData download portal. See this Telegraf installation guide for additional installation instructions and options.
Sign in to the Azure portal.
If you want to migrate classic alert rules and use an existing Linux virtual machine, make sure the virutal machine has a system-assigned identity set to On.
Create a new Linux VM:
Select the Create a resource option from the left-hand navigation pane.
Search for Virtual Machine.
Select Ubuntu 16.04 LTS and select Create.
Provide a VM name like MyTelegrafVM.
Leave the disk type as SSD. Then provide a username, such as azureuser.
For Authentication type, select Password. Then enter a password you'll use later to SSH into this VM.
Choose to Create new resource group. Then provide a name, such as myResourceGroup. Choose your Location. Then select OK.
Select a size for the VM. You can filter by Compute type or Disk type, for example.
On the Settings page in Network > Network Security Group > Select public inbound ports, select HTTP and SSH (22). Leave the rest of the defaults and select OK.
On the summary page, select Create to start the VM deployment.
The VM is pinned to the Azure portal dashboard. After the deployment finishes, the VM summary automatically opens.
In the VM pane, navigate to the Identity tab. Ensure that your VM has a system-assigned identity set to On.
Connect to the VM
Create an SSH connection with the VM. Select the Connect button on the overview page for your VM.
In the Connect to virtual machine page, keep the default options to connect by DNS name over port 22. In Login using VM local account, a connection command is shown. Select the button to copy the command. The following example shows what the SSH connection command looks like:
Paste the SSH connection command into a shell, such as Azure Cloud Shell or Bash on Ubuntu on Windows, or use an SSH client of your choice to create the connection.
Install and configure Telegraf
To install the Telegraf Debian package onto the VM, run the following commands from your SSH session:
# download the package to the VM wget https://dl.influxdata.com/telegraf/releases/telegraf_1.8.0~rc1-1_amd64.deb # install the package sudo dpkg -i telegraf_1.8.0~rc1-1_amd64.deb
Telegraf’s configuration file defines Telegraf’s operations. By default, an example configuration file is installed at the path /etc/telegraf/telegraf.conf. The example configuration file lists all possible input and output plug-ins. However, we'll create a custom configuration file and have the agent use it by running the following commands:
# generate the new Telegraf config file in the current directory telegraf --input-filter cpu:mem --output-filter azure_monitor config > azm-telegraf.conf # replace the example config with the new generated config sudo cp azm-telegraf.conf /etc/telegraf/telegraf.conf
The preceding code enables only two input plug-ins: cpu and mem. You can add more input plug-ins, depending on the workload that runs on your machine. Examples are Docker, MySQL, and NGINX. For a full list of input plug-ins, see the Additional configuration section.
Finally, to have the agent start using the new configuration, we force the agent to stop and start by running the following commands:
# stop the telegraf agent on the VM sudo systemctl stop telegraf # start the telegraf agent on the VM to ensure it picks up the latest configuration sudo systemctl start telegraf
Now the agent will collect metrics from each of the input plug-ins specified and emit them to Azure Monitor.
Plot your Telegraf metrics in the Azure portal
Open the Azure portal.
Navigate to the new Monitor tab. Then select Metrics.
Select your VM in the resource selector.
Select the Telegraf/CPU namespace, and select the usage_system metric. You can choose to filter by the dimensions on this metric or split on them.
The preceding walkthrough provides information on how to configure the Telegraf agent to collect metrics from a few basic input plug-ins. The Telegraf agent has support for over 150 input plug-ins, with some supporting additional configuration options. InfluxData has published a list of supported plugins and instructions on how to configure them.
Additionally, in this walkthrough, you used the Telegraf agent to emit metrics about the VM the agent is deployed on. The Telegraf agent can also be used as a collector and forwarder of metrics for other resources. To learn how to configure the agent to emit metrics for other Azure resources, see Azure Monitor Custom Metric Output for Telegraf.
Clean up resources
When they're no longer needed, you can delete the resource group, virtual machine, and all related resources. To do so, select the resource group for the virtual machine and select Delete. Then confirm the name of the resource group to delete.
- Learn more about custom metrics.