Troubleshoot access, ingestion, and operation of your Azure Data Explorer cluster in your virtual network

In this section you learn how to troubleshoot connectivity, operational, and cluster creation issues for a cluster that is deployed into your Virtual Network.

Access issues

If you have an issue while accessing cluster using the public (cluster.region.kusto.windows.net) or private (private-cluster.region.kusto.windows.net) endpoint and you suspect it's related to virtual network setup, perform the following steps to troubleshoot the issue.

Check TCP connectivity

The first step includes checking TCP connectivity using Windows or Linux OS.

  1. Download TCping to the machine connecting to the cluster.

  2. Ping the destination from the source machine by using the following command:

    C:\> tcping -t yourcluster.kusto.windows.net 443 
    ** Pinging continuously.  Press control-c to stop **
    Probing 1.2.3.4:443/tcp - Port is open - time=100.00ms
    

If the test isn't successful, proceed with the following steps. If the test is successful, the issue isn't due to a TCP connectivity issue. Go to operational issues to troubleshoot further.

Check the Network Security Group (NSG)

Check that the Network Security Group (NSG) attached to the cluster's subnet, has an inbound rule that allows access from the client machine's IP for port 443.

Check route table

If the cluster's subnet has force-tunneling setup to firewall (subnet with a route table that contains the default route '0.0.0.0/0'), make sure that the machine IP address has a route with next hop type to VirtualNetwork/Internet. This route is required to prevent asymmetric route issues.

Ingestion issues

If you're experiencing ingestion issues and you suspect it's related to virtual network setup, perform the following steps.

Check ingestion health

Check that the cluster ingestion metrics indicate a healthy state.

Check security rules on data source resources

If the metrics indicate that no events were processed from the data source (Events processed metric for Event/IoT Hubs), make sure that the data source resources (Event Hub or Storage) allow access from cluster's subnet in the firewall rules or service endpoints.

Check security rules configured on cluster's subnet

Make sure cluster's subnet has NSG, UDR, and firewall rules are properly configured. In addition, test network connectivity for all dependent endpoints.

Cluster creation and operations issues

If you're experiencing cluster creation or operation issues and you suspect it's related to virtual network setup, follow these steps to troubleshoot the issue.

Check the "DNS servers" configuration

Custom DNS server aren't supported. Use the default option in the DNS servers configuration section of your Virtual Network.

Diagnose the virtual network with the REST API

The ARMClient is used to call the REST API using PowerShell.

  1. Log in with ARMClient

    armclient login
    
  2. Invoke diagnose operation

    $subscriptionId = '<subscription id>'
    $clusterName = '<name of cluster>'
    $resourceGroupName = '<resource group name>'
    $apiversion = '2019-11-09'
    
    armclient post "https://management.azure.com/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.Kusto/clusters/$clusterName/diagnoseVirtualNetwork?api-version=$apiversion" -verbose
    
  3. Check the response

    HTTP/1.1 202 Accepted
    ...
    Azure-AsyncOperation: https://management.azure.com/subscriptions/{subscription-id}/providers/Microsoft.Kusto/locations/{location}/operationResults/{operation-id}?api-version=2019-11-09
    ...
    
  4. Wait for operation completion

    armclient get https://management.azure.com/subscriptions/$subscriptionId/providers/Microsoft.Kusto/locations/{location}/operationResults/{operation-id}?api-version=2019-11-09
    
    {
      "id": "/subscriptions/{subscription-id}/providers/Microsoft.Kusto/locations/{location}/operationresults/{operation-id}",
      "name": "{operation-name}",
      "status": "[Running/Failed/Completed]",
      "startTime": "{start-time}",
      "endTime": "{end-time}",
      "properties": {...}
    }
    

    Wait until the status property shows Completed, then the properties field should show:

    {
      "id": "/subscriptions/{subscription-id}/providers/Microsoft.Kusto/locations/{location}/operationresults/{operation-id}",
      "name": "{operation-name}",
      "status": "Completed",
      "startTime": "{start-time}",
      "endTime": "{end-time}",
      "properties": {
        "Findings": [...]
      }
    }
    

If the Findings property shows an empty result, it means that all network tests passed and no connections are broken. If the following error is shown, Outbound dependency '{dependencyName}:{port}' might be not satisfied (Outbound), the cluster can't reach the dependent service endpoints. Proceed with the following steps.

Check Network Security Group (NSG)

Make sure that the Network Security Group is configured properly per the instructions in Dependencies for VNet deployment

Check route table

If the cluster's subnet has force-tunneling set up to firewall (subnet with a route table that contains the default route '0.0.0.0/0') make sure that the management IP addresses) and health monitoring IP addresses have a route with next hop type Internet, and source address prefix to 'management-ip/32' and 'health-monitoring-ip/32'. This route required to prevent asymmetric route issues.

Check firewall rules

If you force tunnel subnet outbound traffic to a firewall, make sure all dependencies FQDN (for example, .blob.core.windows.net) are allowed in the firewall configuration as described in securing outbound traffic with firewall.