Edit

Azure Cosmos DB SDK observability

  • Article

APPLIES TO: NoSQL

The Azure Cosmos DB .NET and Java SDKs support distributed tracing to help you monitor your applications. Tracing the flow of requests is helpful in debugging, analyzing latency and performance, and gathering diagnostics. Instrument tracing for your applications using OpenTelemetry, which is vendor-neutral and has a set of semantic conventions to ensure a standardized data format regardless of your chosen exporter, or use the Application Insights SDK or Azure Monitor OpenTelemetry Distro.

Get started

Distributed tracing is available in the following SDKs:

SDK Supported version Notes
.NET v3 SDK >= 3.36.0 This feature is available in both preview and non-preview versions. For non-preview versions, it's off by default. You can enable tracing by setting DisableDistributedTracing = false in CosmosClientOptions.CosmosClientTelemetryOptions.
.NET v3 SDK preview >= 3.33.0-preview This feature is available in both preview and non-preview versions. For preview versions, it's on by default. You can disable tracing by setting DisableDistributedTracing = true in CosmosClientOptions.CosmosClientTelemetryOptions.
Java v4 SDK >= 4.43.0

Trace attributes

Azure Cosmos DB traces follow the OpenTelemetry database specification and also provide several custom attributes. You can see different attributes depending on the operation of your request, and these attributes are core attributes for all requests.

Attribute Type Description
net.peer.name string Azure Cosmos DB host name.
db.name string Azure Cosmos DB database name.
db.system string Identifier for the database service. Is cosmosdb for all requests.
db.operation string Operation name, ex. CreateItemAsync.
db.cosmosdb.container string Azure Cosmos DB container name.
db.cosmosdb.client_id string Identifier representing a unique client instance.
db.cosmosdb.operation_type string Operation type, ex. Create.
db.cosmosdb.connection_mode string Client connection mode. Either direct or gateway.
db.cosmosdb.status_code int Status code for the request.
db.cosmosdb.sub_status_code int Sub status code for the request.
db.cosmosdb.request_charge double RUs consumed for the operation.
db.cosmosdb.regions_contacted string List of regions contacted in the Azure Cosmos DB account.
user_agent.original string Full user-agent string generated by the Azure Cosmos DB SDK.

Gather diagnostics

If you configured logs in your trace provider, you can automatically get diagnostics for Azure Cosmos DB requests that failed or had high latency. These logs can help you diagnose failed and slow requests without requiring any custom code to capture them.

In addition to getting diagnostic logs for failed requests, you can configure different latency thresholds for when to collect diagnostics from successful requests. The default values are 100 ms for point operations and 500 ms for non point operations. These thresholds can be adjusted through client options.

CosmosClientOptions options = new CosmosClientOptions()
{
    CosmosClientTelemetryOptions = new CosmosClientTelemetryOptions()
    {
        DisableDistributedTracing = false,
        CosmosThresholdOptions = new CosmosThresholdOptions()
        {
            PointOperationLatencyThreshold = TimeSpan.FromMilliseconds(100),
            NonPointOperationLatencyThreshold = TimeSpan.FromMilliseconds(500)
        }
    },
};

You can configure the log level to control which diagnostics logs you receive.

Log Level Description
Error Logs for errors only.
Warning Logs for errors and high latency requests based on configured thresholds.
Information There are no specific information level logs. Logs in this level are the same as using Warning.

Depending on your application environment, there are different ways to configure the log level. Here's a sample configuration in appSettings.json:

{ 
    "Logging": {​
        "LogLevel": {​
            "Azure-Cosmos-Operation-Request-Diagnostics": "Information"​
        }​
    }
}

Configure OpenTelemetry

To use OpenTelemetry with the Azure Cosmos DB SDKs, add the Azure.Cosmos.Operation source to your trace provider. OpenTelemetry is compatible with many exporters that can ingest your data. The following sample uses the Azure Monitor OpenTelemetry Exporter, but you can choose to configure any exporter you wish. Depending on your chosen exporter, you might see a delay ingesting data of up to a few minutes.

Tip

If you use the Azure.Monitor.OpenTelemetry.Exporter package, ensure you're using version >= 1.0.0-beta.11. If you're using ASP.NET Core and Azure Monitor, we recommend using the Azure Monitor OpenTelemetry Distro instead.

This sample shows how to configure OpenTelemetry for a .NET console app. See the complete sample on GitHub.

ResourceBuilder resource = ResourceBuilder.CreateDefault().AddService(
            serviceName: serviceName,
            serviceVersion: "1.0.0");

// Set up logging to forward logs to chosen exporter
using ILoggerFactory loggerFactory
    = LoggerFactory.Create(builder => builder
                                        .AddConfiguration(configuration.GetSection("Logging"))
                                        .AddOpenTelemetry(options =>
                                        {
                                            options.IncludeFormattedMessage = true;
                                            options.SetResourceBuilder(resource);
                                            options.AddAzureMonitorLogExporter(o => o.ConnectionString = aiConnectionString); // Set up exporter of your choice
                                        }));
/*.AddFilter(level => level == LogLevel.Error) // Filter  is irrespective of event type or event name*/

AzureEventSourceLogForwarder logforwader = new AzureEventSourceLogForwarder(loggerFactory);
logforwader.Start();

// Configure OpenTelemetry trace provider
AppContext.SetSwitch("Azure.Experimental.EnableActivitySource", true);
_traceProvider = Sdk.CreateTracerProviderBuilder()
    .AddSource("Azure.Cosmos.Operation", // Cosmos DB source for operation level telemetry
               "Sample.Application") 
    .AddAzureMonitorTraceExporter(o => o.ConnectionString = aiConnectionString) // Set up exporter of your choice
    .AddHttpClientInstrumentation() // Added to capture HTTP telemetry
    .SetResourceBuilder(resource)
    .Build();

Configure the Application Insights SDK

There are many different ways to configure Application Insights depending on the language your application is written in and your compute environment. For more information, see the Application Insights documentation. Ingestion of data into Application Insights can take up to a few minutes.

Note

Use version >= 2.22.0-beta2 of the Application Insights package for your target .NET environment.

The following sample shows how to configure Application Insights for a .NET console app. See the complete sample on GitHub.

IServiceCollection services = new ServiceCollection();
services.AddApplicationInsightsTelemetryWorkerService((ApplicationInsightsServiceOptions options) => options.ConnectionString = aiConnectionString);

IServiceProvider serviceProvider = services.BuildServiceProvider();
telemetryClient = serviceProvider.GetRequiredService<TelemetryClient>();

Once trace data is ingested into Application Insights, you can visualize it in the Azure portal to understand the request flow in your application. Here's an example of trace data from a cross partition query in the transaction search in the left navigation of the Azure portal.

Screenshot of distributed tracing of an Azure Cosmos DB cross-partition query in the Application Insights transaction search.

Next Steps