Trouble Viewing Custom Metrics in Azure Monitor via OpenTelemetry AzureMonitorMetricExporter Using Python

Question

I've been attempting to integrate OpenTelemetry with Azure Monitor to track custom metrics from my Python application, specifically using the AzureMonitorMetricExporter to send metrics to Azure Monitor (Application Insights). Despite my efforts and following the setup guides, my custom metrics are not visible in the Azure Portal's Metrics Explorer under my Application Insights resource.

Environment:

• Python 3.8 application, deployed as Azure Functions.

Key Packages:

• opentelemetry-api
• opentelemetry-sdk
• opentelemetry-exporter-azuremonitor I've been attempting to integrate OpenTelemetry with Azure Monitor to track custom metrics from my Python application, specifically using the AzureMonitorMetricExporter to send metrics to Azure Monitor (Application Insights). Despite my efforts and following the setup guides, my custom metrics are not visible in the Azure Portal's Metrics Explorer under my Application Insights resource. Environment:
  • Python 3.8 application, deployed as Azure Functions.

Key Packages:

• opentelemetry-api
• opentelemetry-sdk
• opentelemetry-exporter-azuremonitor

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from azure.monitor.opentelemetry.exporter import AzureMonitorMetricExporter
import os

# Set the Meter Provider
metrics.set_meter_provider(MeterProvider())

# Initialize the Azure Monitor Metrics Exporter
exporter = AzureMonitorMetricExporter.from_connection_string(
    os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)

# Get a Meter
meter = metrics.get_meter("100x-functions", version="1.0")

# Create a counter metric
requests_counter = meter.create_counter(
    name="requests",
    description="Number of requests",
    unit="1",
)

# Record a metric to track requests
requests_counter.add(1, {"environment": "production"})

Despite these configurations, when I check the Metrics Explorer in the Azure Portal, I don't see my custom metrics listed under any namespace. I've double-checked the connection string for accuracy and ensured there are no visible errors in the application logs regarding metric export.

Questions:

• Could I be missing a step in my setup that's preventing the metrics from being visible in Azure Monitor?
• Is there a common issue or configuration detail related to the AzureMonitorMetricExporter and OpenTelemetry that I might have overlooked?
• Any recommendations on troubleshooting steps or additional configurations to ensure my custom metrics are exported and visible in Azure Monitor?

I appreciate any insights or suggestions the community can offer on resolving this issue.

Answer 1

@noob I tried to work through this issue and the following are some of the observations based on your code snippet provided. (note - I worked on a sample repro using the Azure function development through VSCode locally, but the concept should also apply to deployed Azure Function).

I see a number of deviations from the documented sample as available here - Metric instrument usage.

I used the code snippet provided in the question and the documented sample, and the following worked for me where customMetric is being tracked for a HTTP triggered Azure Function written in Python. I have added comments to the updated lines of code.:

import azure.functions as func
import logging
from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from azure.monitor.opentelemetry.exporter import AzureMonitorMetricExporter
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader #-----UPDATED-------
import os

app = func.FunctionApp(http_auth_level=func.AuthLevel.ANONYMOUS)

exporter = AzureMonitorMetricExporter.from_connection_string(os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])


reader = PeriodicExportingMetricReader(exporter, export_interval_millis=5000)  #-----UPDATED---
meter_provider = MeterProvider(metric_readers=[reader]) #-----UPDATED-------
metrics.set_meter_provider(meter_provider) #-----UPDATED-------

  
meter = metrics.get_meter_provider().get_meter("sample") #-----UPDATED-------

    # Create a counter metric
requests_counter = meter.create_counter(
    name="requests",
    description="Number of requests",
    unit="1")

@app.route(route="HTTPTriggerFunc")
def HTTPTriggerFunc(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    #open telemetry start *****************************************************

    # Record a metric to track requests
    requests_counter.add(1, {"environment": "production"})
        
    meter_provider.force_flush() #-----UPDATED----
    
    #---open telemetry end *****************************************************

    name = req.params.get('name')
    if not name:
        try:
            req_body = req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    if name:
        return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed successfully.")
    else:
        return func.HttpResponse(
             "This HTTP triggered function executed successfully. Pass a name in the query string or in the request body for a personalized response.",
             status_code=200
        )

While most of the code in the snippet above is boiler plate code as available when creating a HTTP trigger python-based Azure function, the lines marked with #-----UPDATED------- shows the updates made in addition to what you had provided in the question.

After sending a couple of telemetry, I could see the metrics in customMetrics table in ApplicationInsights Workspace using the query below:

customMetrics
| where customDimensions == '{"environment":"production"}'

And also, metrics explorer had this listed under the azure.applicationinsights Metric namespace

I hope the sample code snippet shared will also answer the queries you had.

Please let me know if you have any questions.

If the answer helped, please click Accept answer so that it can help others in the community looking for help on similar topics.

Answer 2

Hello noob ,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Sequel to your questions, I understand that your are facing difficulties in viewing custom metrics in Azure Monitor via OpenTelemetry AzureMonitorMetricExporter using Python. Despite configuring the exporter and sending custom metrics from a Python application deployed as Azure Functions, the metrics are not visible in the Azure Portal's Metrics Explorer under the Application Insights resource.

Your questions are the followings:

1. Could there be a missing step in the setup preventing the metrics from being visible in Azure Monitor?
2. Is there a common issue or configuration detail related to AzureMonitorMetricExporter and OpenTelemetry that might have been overlooked?
3. What are the recommended troubleshooting steps or additional configurations to ensure custom metrics are exported and visible in Azure Monitor?

Based on what you have done:

Firstly, initializing the Meter Provider and Exporter:

from opentelemetry import metrics
from opentelemetry.sdk.metrics import MeterProvider
from azure.monitor.opentelemetry.exporter import AzureMonitorMetricExporter
import os
# Set the Meter Provider
metrics.set_meter_provider(MeterProvider())
# Initialize the Azure Monitor Metrics Exporter
exporter = AzureMonitorMetricExporter.from_connection_string(
    os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
)

Analysis:

• The MeterProvider and AzureMonitorMetricExporter are initialized correctly.
• The connection string is retrieved from the environment variable APPLICATIONINSIGHTS_CONNECTION_STRING.

Suggested Step:

Verify that the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable is correctly set in the Azure Functions environment and that it contains the valid connection string for the Application Insights resource.

Secondly, Creating and Recording Custom Metrics:

# Get a Meter
meter = metrics.get_meter("100x-functions", version="1.0")
# Create a counter metric
requests_counter = meter.create_counter(
    name="requests",
    description="Number of requests",
    unit="1",
)
# Record a metric to track requests
requests_counter.add(1, {"environment": "production"})

Analysis:

• A custom counter metric named "requests" is created and recorded with a value of 1.
• The metric is associated with the environment tag "production".

Suggested Step:

Ensure that the recorded metric values are being sent to the Azure Monitor exporter correctly.

Check if the metric namespace and tags are configured appropriately to appear in the Metrics Explorer.

Now, back to answer your questions:

Q1. Yes, there could potentially be a missing step in the setup that is preventing the metrics from being visible in Azure Monitor. Some common reasons for metrics not appearing in Azure Monitor despite correct configuration include and not limited to the followings:

• Missing Configuration for Metric Exporter.
• Incomplete Metric Recording.
• Incorrect Namespace or Tags.
• Permissions and Access Control.
• Network Restrictions.

By carefully reviewing the setup and addressing any potential missing steps or misconfigurations, the visibility of custom metrics in Azure Monitor can be improved.

Q2. Yes, a common issue or configuration detail related to AzureMonitorMetricExporter and OpenTelemetry that might have been overlooked is the compatibility of the exporter version with the installed OpenTelemetry SDK version.

It's essential to ensure that the versions of the AzureMonitorMetricExporter and the OpenTelemetry SDK packages are compatible with each other. Incompatibility between these versions can lead to issues with metric export or functionality gaps that might result in custom metrics not being visible in Azure Monitor.

Additionally, overlooking the configuration of the exporter, such as authentication mechanisms or connection settings, can also be a common oversight. Misconfigured exporters may fail to send metrics data to Azure Monitor, resulting in the metrics not being visible in the Metrics Explorer.

Lastly, overlooking the requirements for setting up and configuring Azure Monitor itself, such as proper linking of Application Insights resources or enabling data ingestion, can also lead to metrics visibility issues despite correct configuration of the exporter and OpenTelemetry SDK.

Q3. To troubleshoot and ensure that custom metrics are exported and visible in Azure Monitor, these are the recommended steps:

1. Confirm that the APPLICATIONINSIGHTS_CONNECTION_STRING environment variable is correctly set in the Azure Functions environment. Ensure it contains the valid connection string for the Application Insights resource. Check for any typos or mistakes in the connection string. Review the Azure Monitor exporter configuration to ensure it's properly initialized and configured.
2. Verify that custom metrics are being recorded properly within the application code. Add logging or debugging mechanisms to track the recording of custom metrics and ensure they have the expected values.

   1.       # Example code for logging metric recording
            import logging
            logging.info("Recording custom metric: requests_counter")
            requests_counter.add(1, {"environment": "production"})
            
      

3. Ensure that custom metrics are associated with the correct namespace and tags. Review the metric creation code to confirm the namespace and tags are configured appropriately.

      # Example code for creating custom metric with namespace and tags
      requests_counter = meter.create_counter(
          name="requests",
          description="Number of requests",
          unit="1",
          labels={"environment": "production"}
      )
      
   

Additionally:

1. Verify the configuration of Azure Monitor in the Azure Portal. Confirm that the Application Insights resource is correctly linked to the Azure Functions application. Ensure that data ingestion is enabled for the Application Insights resource.
2. Check for any logs or debug output from the Azure Monitor exporter. Look for any errors or exceptions encountered during metric export.
3. Review the additional resources, training, and documentation for AzureMonitorMetricExporter and OpenTelemetry for any specific troubleshooting steps or common issues by the right side of this page.

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

Please remember to "Accept Answer" if answer helped, so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam