Troubleshoot latency using Storage Analytics logs

Diagnosing and troubleshooting are key skills for building and supporting client applications with Azure Storage.

Because of the distributed nature of an Azure application, diagnosing and troubleshooting both errors and performance issues may be more complex than in traditional environments.

The following steps demonstrate how to identify and troubleshoot latency issues using Azure Storage Analytics logs and optimize the client application.

  1. Download the Storage Analytics logs.

  2. Use the following PowerShell script to convert the raw format logs into tabular format:

    $Columns = 
         (   "version-number",
             "request-start-time",
             "operation-type",
             "request-status",
             "http-status-code",
             "end-to-end-latency-in-ms",
             "server-latency-in-ms",
             "authentication-type",
             "requester-account-name",
             "owner-account-name",
             "service-type",
             "request-url",
             "requested-object-key",
             "request-id-header",
             "operation-count",
             "requester-ip-address",
             "request-version-header",
             "request-header-size",
             "request-packet-size",
             "response-header-size",
             "response-packet-size",
             "request-content-length",
             "request-md5",
             "server-md5",
             "etag-identifier",
             "last-modified-time",
             "conditions-used",
             "user-agent-header",
             "referrer-header",
             "client-request-id"
         )
    
    $logs = Import-Csv "REPLACE THIS WITH FILE PATH" -Delimiter ";" -Header $Columns
    
    $logs | Out-GridView -Title "Storage Analytic Log Parser"
    
  3. The script will launch a GUI window where you can filter the information by columns, as shown below.

    Screenshot of the Storage Analytic Log Parser Window.

  4. Narrow down the log entries based on operation-type, and look for the log entry created during the issue's time frame.

    Screenshot of the operation-type log entries.

  5. During the time when the issue occurred, the following values are important:

    • Operation-type = GetBlob
    • request-status = SASNetworkError
    • End-to-End-Latency-In-Ms = 8,453
    • Server-Latency-In-Ms = 391

    End-to-End Latency is calculated using the following equation:

    • End-to-End Latency = Server-Latency + Client Latency

    Calculate the Client Latency using the log entry:

    • Client Latency = End-to-End LatencyServer-Latency

      Example: 8,453 – 391 = 8,062 ms

    The following table provides information about the high-latency OperationType and RequestStatus results:

    Blob Type RequestStatus=
    Success
    RequestStatus=
    (SAS)NetworkError
    Recommendation
    GetBlob Yes No GetBlob Operation: RequestStatus = Success
    GetBlob No Yes GetBlob Operation: RequestStatus = (SAS)NetworkError
    PutBlob Yes No Put Operation: RequestStatus = Success
    PutBlob No Yes Put Operation: RequestStatus = (SAS)NetworkError

Status results

GetBlob Operation: RequestStatus = Success

Check the following values as mentioned in step 5 of the Recommended steps section:

  • End-to-End Latency
  • Server-Latency
  • Client-Latency

In a GetBlob Operation with RequestStatus = Success, if Max Time is spent in Client-Latency, this indicates that Azure Storage is spending a large volume of time writing data to the client. This delay indicates a Client-Side issue.

Recommendation:

  • Investigate the code in your client.
  • Use Wireshark, Microsoft Message Analyzer, or Tcping to investigate network connectivity issues from the client.

GetBlob Operation: RequestStatus = (SAS)NetworkError

Check the following values as mentioned in step 5 of the Recommended steps section:

  • End-to-End Latency
  • Server-Latency
  • Client-Latency

In a GetBlob Operation with RequestStatus = (SAS)NetworkError, if Max Time is spent in Client-Latency, the most common issue is that the client is disconnecting before a timeout expires in the storage service.

Recommendation:

  • Investigate the code in your client to understand why and when the client disconnects from the storage service.
  • Use Wireshark, Microsoft Message Analyzer, or Tcping to investigate network connectivity issues from the client.

Put Operation: RequestStatus = Success

Check the following values as mentioned in step 5 of the Recommended steps section:

  • End-to-End Latency
  • Server-Latency
  • Client-Latency

In a Put Operation with RequestStatus = Success, if Max Time is spent in Client-Latency, this indicates that the Client is taking more time to send data to the Azure Storage. This delay indicates a Client-Side Issue.

Recommendation:

  • Investigate the code in your client.
  • Use Wireshark, Microsoft Message Analyzer, or Tcping to investigate network connectivity issues from the client.

Put Operation: RequestStatus = (SAS)NetworkError

Check the following values as mentioned in step 5 of the "Recommended steps" section:

  • End-to-End Latency
  • Server-Latency
  • Client-Latency

In a PutBlob Operation with RequestStatus = (SAS)NetworkError, if Max Time is spent in Client-Latency, the most common issue is that the client is disconnecting before a timeout expires in the storage service.

Recommendation:

  • Investigate the code in your client to understand why and when the client disconnects from the storage service.
  • Use Wireshark, Microsoft Message Analyzer, or Tcping to investigate network connectivity issues from the client.

Third-party information disclaimer

The third-party products that this article discusses are manufactured by companies that are independent of Microsoft. Microsoft makes no warranty, implied or otherwise, about the performance or reliability of these products.

Contact us for help

If you have questions or need help, create a support request, or ask Azure community support. You can also submit product feedback to Azure feedback community.