Connect to Azure Data Explorer from Azure Databricks

05/21/2020
2 minutes to read

Azure Databricks is an Apache Spark-based analytics platform that's optimized for the Microsoft Azure platform. This article shows you how to use Azure Databricks to access data from Azure Data Explorer. There are several ways to authenticate with Azure Data Explorer, including a device login and an Azure Active Directory (Azure AD) app.

Prerequisites

Create an Azure Data Explorer cluster and database.
Create an Azure Databricks workspace. Under Azure Databricks Service, in the Pricing Tier drop-down list, select Premium. This selection enables you to use Azure Databricks secrets to store your credentials and reference them in notebooks and jobs.
Create a cluster in Azure Databricks with the default settings.

Install the Kusto Spark connector on your Azure Databricks cluster

To install the spark-kusto-connector on your Azure Databricks cluster:

Go to your Azure Databricks workspace and create a library.
Search for the spark-kusto-connector package on Maven Central, install the latest version, and attach to your cluster.

Connect to Azure Data Explorer by using a device authentication

Sample code.

Connect to Azure Data Explorer by using an Azure AD app

Create Azure AD app by provisioning an Azure AD application.
Grant access to your Azure AD app in your Azure Data Explorer database as follows:
```
.set database <DB Name> users ('aadapp=<AAD App ID>;<AAD Tenant ID>') 'AAD App to connect Spark to ADX
```
Parameter Description

DB Name your database name

AAD App ID your Azure AD app ID

AAD Tenant ID your Azure AD tenant ID

Parameter	Description
`DB Name`	your database name
`AAD App ID`	your Azure AD app ID
`AAD Tenant ID`	your Azure AD tenant ID

Find your Azure AD tenant ID

To authenticate an application, Azure Data Explorer uses your Azure AD tenant ID. To find your tenant ID, use the following URL. Substitute your domain for YourDomain.

https://login.windows.net/<YourDomain>/.well-known/openid-configuration/

For example, if your domain is contoso.com, the URL is: https://login.windows.net/contoso.com/.well-known/openid-configuration/. Select this URL to see the results. The first line is as follows:

"authorization_endpoint":"https://login.windows.net/6babcaad-604b-40ac-a9d7-9fd97c0b779f/oauth2/authorize"

Your tenant ID is 6babcaad-604b-40ac-a9d7-9fd97c0b779f.

Store and secure your Azure AD app ID and key (optional)

Store and secure your Azure AD app ID and key by using Azure Databricks secrets as follows:

Set up the CLI.
Install the CLI.
Set up authentication.
Configure the secrets by using the following sample commands:

databricks secrets create-scope --scope adx

databricks secrets put --scope adx --key myaadappid

databricks secrets put --scope adx --key myaadappkey

databricks secrets list --scope adx

Sample Code

Sample code.
Update the placeholder values with your cluster name, database name, table name, Azure AD tenant ID, AAD App ID, and AAD App Key. If you are storing your credentials in databricks secrets store, update the code accordingly to retrieve values from dbutils.