Create a private endpoint for Azure Data Explorer

Use an Azure private endpoint to connect to your cluster in your Azure virtual network (VNet).

Private endpoints use private IP addresses from your virtual network to connect you privately to your cluster. With this setup, network traffic between a client on your private network and the cluster travels over the virtual network and a private link on the Microsoft backbone network. This article shows you how to create a private endpoint for your cluster.

Prerequisites

Create a private endpoint

There are several ways to create a private endpoint for a cluster.

Create a private endpoint during the deployment of your cluster in the portal

Use the following information to create a private endpoint whilst creating your cluster.

  1. In the Create an Azure Data Explorer cluster page, select the Network tab.

  2. Under Connectivity method, select Private Endpoints.

  3. Under Private Endpoint, select Add.

    Screenshot of the cluster creation page, showing the private endpoint option during deployment.

  4. Configure the private endpoint.

  5. Complete the steps to create the cluster.

Create a private endpoint on an existing cluster

Use the following information to create a private endpoint on an existing cluster.

  1. In the Azure portal, navigate to your cluster and then select Networking.

  2. Select Private endpoint connections, and then select + Private endpoint.

    Screenshot of the networking page, showing the first step in the creation of a private endpoint.

  3. Configure the private endpoint.

Configure your private endpoint

  1. On the Basics tab, fill out the basic cluster details with the following information, and then select on Next.

    Screenshot of the create private endpoint page, showing the basic information.

    Setting Suggested value Field description
    Subscription Your subscription Select the Azure subscription to use for your private endpoint
    Resource group Your resource group Use an existing resource group or create a new resource group
    Name myNewPrivateEndpoint Choose a name that identifies your Private Endpoint in the resource group
    Region (Europe) West Europe Select the region that best meets your needs
  2. On the Resources tab, select Connect to an Azure resource in my directory, fill out the resource details with the following information, and then select Next.

    Screenshot of the create private endpoint page, showing the resources information.

    Setting Suggested value Field description
    Subscription Your subscription Select the Azure subscription that you want to use for your cluster
    Resource type Your resource group Select "Microsoft.Kusto/clusters
    Resource contoso-adx Choose the cluster that should be used as the destination for the new Azure Private Endpoint
    Target sub-resource cluster There's no other option

    Alternatively, you can select Connect to an Azure resource by resource ID or alias. This enables you to create a private endpoint to a cluster in another tenant or if you don't have at least Reader access on the resource.

    Setting Suggested value Field description
    ResourceId or alias /subscriptions/... The resource ID or alias that someone has shared with you. The easiest way to get the resource ID is to navigate to the cluster in the Azure portal and copy the Resource ID from the Properties sections
    Target sub-resource cluster There's no other option
    Request message Please approve The resource owner sees this message while managing private endpoint connection
  3. On the Virtual Network tab, under Networking, specify the Virtual Network and Subnet where you want to deploy the private endpoint.

  4. Under Private IP configuration, select Dynamically allocate IP address.

    Note

    The Statically allocate IP address option is not supported.

  5. Under Private DNS integration, turn on the Integrate with the private DNS zone. It's needed to resolve the engine and data management endpoints including the storage accounts required for ingestion and export features.

    Note

    We recommend that you use the Private DNS integration option. If you have a situation where you can't use the option, follow the instructions under Use a custom DNS server.

  6. Select Next.

    Screenshot of the create private endpoint page, showing the virtual network configuration.

  7. On the Tags tab, configure any tags you require, and then select Next.

  8. Review the configuration details and then select Create to create the private endpoint resource.

    Screenshot of the create private endpoint page, showing the review and create summary.

Verify the private endpoint creation

Once the creation of the private endpoint is complete, you'll be able to access it in the Azure portal.

Screenshot of the create private endpoint page, showing the results of the private endpoint creation.

To see all the private endpoints created for your cluster:

  1. In the Azure portal, navigate to your cluster and then select Networking

  2. Select Private endpoint. In the table, you can see all private endpoints created for your cluster.

    Screenshot of the networking page, showing the all private endpoints of the cluster in the Azure portal.

  3. In the Connection state column, verify that the private endpoint is approved.

Use a custom DNS server

In some situations, you may not be able to integrate with the private DNS zone of the virtual network. For example, you may be using your own DNS server or you create DNS records using the host files on your virtual machines. This section describes how to get to the DNS zones.

  1. Install choco

  2. Install ARMClient

    choco install armclient
    
  3. Log in with ARMClient

    armclient login
    
  4. Run the following REST API call to get the private DNS zones for your cluster:

    #replace the <...> placeholders with the correct values
    armclient GET /subscriptions/<subscriptionIdADX>/resourceGroups/<resourceGroupNameADX>/providers/Microsoft.Kusto/clusters/<clusterName>/privateLinkResources?api-version=2022-02-01
    
  5. Check the response. The required DNS zones are in the "requiredZoneNames" array in the response of the result.

    {
      "value": [
        {
          "id": "/subscriptions/<subscriptionIdADX>/resourceGroups/<resourceGroupNameADX>/providers/Microsoft.Kusto/Clusters/<clusterName>/PrivateLinkResources/cluster",
          "name": "<clusterName>/cluster",
          "type": "Microsoft.Kusto/Clusters/PrivateLinkResources",
          "location": "<the region of your cluster>",
          "properties": {
            "groupId": "cluster",
            "requiredMembers": [
              "Engine",
              "DataManagement",
              "blob-storageAccount1",
              "queue-storageAccount1",
              "table-storageAccount1",
              "blob-storageAccount2",
              "queue-storageAccount2",
              "table-storageAccount2"
            ],
            "requiredZoneNames": [
              "privatelink.<the region of your cluster>.kusto.windows.net",
              "privatelink.blob.core.windows.net",
              "privatelink.queue.core.windows.net",
              "privatelink.table.core.windows.net"
            ],
            "provisioningState": "Succeeded"
          }
        }
      ]
    }
    
  6. in the Azure portal, navigate to your private endpoint, and select DNS configuration. On this page, you can get the required information for the IP address mapping to the DNS name.

    Screenshot of the DNS configuration page, showing the DNS configuration of the private endpoint.

    Warning

    This information allows you to propagate your custom DNS server with the necessary records. We highly recommend that you integrate with the private DNS Zones of the virtual network and don't configure your own custom DNS server. The nature of private endpoints for Azure Data Explorer clusters is different than for other Azure PaaS services. In some situations, such as high ingestion loads, in order to increase throughput it might be necessary for the service to scale out the number of storage accounts that are accessible via the private endpoint. If you choose to propagate your own custom DNS server, it is your responsibility to take care of updating the DNS records in such situations, and later removing records i the number of storage accounts is scaled back in.