Edit

Integrate elastic logs with Azure Monitor

  • Article

In this article, you'll learn how to start collecting Elasticsearch logs for your Azure Data Manager for Energy instances in Azure Monitor. This integration feature is developed to help you debug Elasticsearch related issues inside Azure Monitor.

Prerequisites

  • You need to have a Log Analytics workspace. It will be used to query the Elasticsearch logs dataset using the Kusto Query Language (KQL) query editor in the Log Analytics workspace. Create a log Analytics workspace in Azure portal.

  • You need to have a storage account. It will be used to store JSON dumps of Elasticsearch & Elasticsearch Operator logs. The storage account doesn’t have to be in the same subscription as your Log Analytics workspace.

Enabling Diagnostic Settings to collect logs in a storage account & a Log Analytics workspace

Every Azure Data Manager for Energy instance comes inbuilt with a managed Elasticsearch service. We collect Elasticsearch logs for internal troubleshooting and debugging purposes. You can get access to these logs by integrating Elasticsearch logs with Azure Monitor.

Each diagnostic setting has three basic parts:

Part Description
Name This is the name of the diagnostic log. Ensure a unique name is set for each log.
Categories Category of logs to send to each of the destinations. The set of categories will vary for each Azure service. Visit: Supported Resource Log Categories
Destinations One or more destinations to send the logs. All Azure services share the same set of possible destinations. Each diagnostic setting can define one or more destinations but no more than one destination of a particular type. It should be a storage account, an Event Hubs namespace or an event hub.

We support two destinations for your Elasticsearch logs from Azure Data Manager for Energy instance:

  • Storage account
  • Log Analytics workspace

Steps to enable diagnostic setting to collect Elasticsearch logs

  1. Open Azure Data Manager for Energy overview page

  2. Select Diagnostic Settings from the left panel

    Screenshot for diagnostic settings overview page. It shows the list of existing settings as well as the option to create a new diagnostic setting.

  3. Select Add diagnostic setting.

  4. Select Elasticsearch logs and Elasticsearch Operator logs under Log categories

  5. Select Send to a Log Analytics workspace

  6. Choose Subscription and the Log Analytics workspace name. You would have created it already as a prerequisite.

    Screenshot for choosing destination settings for Log Analytics workspace. The image shows the subscription and Log Analytics workspace chosen.

  7. Select Archive to storage account

  8. Choose Subscription and storage account name. You would have created it already as a prerequisite. Screenshot that shows choosing destination settings for storage account. Required fields include regions, subscription and storage account.

  9. Select Save.

Go back to the Diagnostic Settings page. You would now see a new diagnostic setting created along with the names of the destination storage account and Log Analytics workspace you chose for this setting.

Screenshot for diagnostic settings overview page. The page shows a sample diagnostic setting to link Elasticsearch logs with Azure Monitor.

View Elasticsearch logs in Log Analytics workspace or download them as JSON files using storage account

How to view & query logs in Log Analytics workspace

The editor in Log Analytics workspace support Kusto (KQL) queries through which you can easily perform complicated queries to extract interesting logs data from the Elasticsearch service running in your Azure Data Manager for Energy instance.

  • Run queries and see Elasticsearch logs in the Log Analytics workspace.

    Screenshot for Elasticsearch logs. The image shows the simplest KQL query that shows all logs in the last 24 hours.

  • Run queries and see Elasticsearch Operator logs in the Log Analytics workspace.

    Screenshot for elasticsearch Operator logs. The image shows the simplest KQL query that shows all logs in the last 24 hours.

How to download logs as JSON files from storage account

  • The Containers menu option in the left panel of your storage account's overview page allows you to browse through the various directories that neatly store your log files.

    Screenshot for storage account that stores elastic logs. The logs can be viewed by selecting 'containers' under the data storage menu-item.

  • Logs are organized into different folders. Drill down by month, date and time.

    Screenshot for JSON file view in storage account. The image shows tracked path from year, month, data, and time to locate a log file.

  • Select any JSON file in your containers to view other options.

    Screenshot to view the downloaded JSON file from storage account. Other options shown include getting a URL for the JSON file.

  • Select Download option to download the JSON file. Open it in a code editor of your choice.

    Screenshot to view downloaded JSON file locally. The images shows formatted logs in Visual Studio Code.

Next steps

After collecting resource logs as explained in this article, there are more capabilities you can explore.