Restrict egress traffic from SQL Server 2019 Big Data Clusters in Azure Kubernetes Service (AKS) private cluster

Article
12/04/2023

Important

The Microsoft SQL Server 2019 Big Data Clusters add-on will be retired. Support for SQL Server 2019 Big Data Clusters will end on February 28, 2025. All existing users of SQL Server 2019 with Software Assurance will be fully supported on the platform and the software will continue to be maintained through SQL Server cumulative updates until that time. For more information, see the announcement blog post and Big data options on the Microsoft SQL Server platform.

You can restrict egress traffic from Big Data Clusters with Azure Kubernetes Service (AKS). The service provisions a standard SKU Load Balancer. This is set up and used for egress by default. Now, the default setup might not meet all scenarios and requirements. For example, if public IPs are disallowed or additional hops are required for egress. You can define a user-defined route (UDR) table if the cluster disallows public IPs and sits behind a network virtual appliance (NVA).

AKS clusters have unrestricted outbound (egress) internet access. This is for management and operational purposes. Worker nodes in an AKS cluster need to access certain ports and fully qualified domain names (FQDNs). The following are examples of this:

When the cluster needs to pull base system container images from Microsoft Container Registry (MCR) during Worker node OS security updates.
When GPU enabled AKS worker nodes need to access endpoints from Nvidia to install a driver.
When customers use AKS work in conjunction with Azure services, such as Azure policy for enterprise-grade compliance, Azure Monitoring (with container insights).
When a Dev Space is enabled, and other similar scenarios.

Note

When you deploy a big data cluster (BDC) in Azure Kubernetes Service (AKS) private cluster, there are no inbound dependencies except for those that are mentioned in this article. You can find all outbound dependencies at control egress traffic for cluster nodes in Azure Kubernetes Service (AKS) .

This article describes how to deploy BDCs in AKS private cluster with advanced networking and UDR. It also explores further integration of BDC with enterprise-grade networking environments.

How to restrict egress traffic with Azure firewall

Azure Firewall provides an Azure Kubernetes Service (AzureKubernetesService) FQDN tag to simplify configuration.

For complete information on the FQDN tag, visit Restrict egress traffic using Azure firewall.

The following image shows how traffic is restricted on an AKS private cluster.

Develop the basic architecture for a Big Data Cluster with Azure Firewall:

Create the resource group & VNet
Create & set up Azure firewall
Create user-defined route table
Set up firewall rules
Create service principal (SP)
Create AKS private cluster
Create BDC deployment profile
Deploy BDC

Create the resource group and VNet

Define a set of environment variables to create resources.

export REGION_NAME=<region>
export RESOURCE_GROUP=private-bdc-aksudr-rg
export SUBNET_NAME=aks-subnet
export VNET_NAME=bdc-vnet
export AKS_NAME=bdcaksprivatecluster

Create the resource group

az group create -n $RESOURCE_GROUP -l $REGION_NAME

Create the VNET

az network vnet create \
  --resource-group $RESOURCE_GROUP \
  --location $REGION_NAME \
  --name $VNET_NAME \
  --address-prefixes 10.0.0.0/8 \
  --subnet-name $SUBNET_NAME \
  --subnet-prefix 10.1.0.0/16

SUBNET_ID=$(az network vnet subnet show \
  --resource-group $RESOURCE_GROUP \
  --vnet-name $VNET_NAME \
  --name $SUBNET_NAME \
  --query id -o tsv)

Create and set up Azure Firewall

Define a set of environment variables for creating resources.

export FWNAME=bdcaksazfw
export FWPUBIP=$FWNAME-ip
export FWIPCONFIG_NAME=$FWNAME-config

az extension add --name azure-firewall

Create a dedicated subnet for the firewall

Note

You can't change the firewall name after creation

az network vnet subnet create \
  --resource-group $RESOURCE_GROUP \
  --vnet-name $VNET_NAME \
  --name AzureFirewallSubnet \
  --address-prefix 10.3.0.0/24

 az network firewall create -g $RESOURCE_GROUP -n $FWNAME -l $REGION_NAME --enable-dns-proxy true

 az network public-ip create -g $RESOURCE_GROUP -n $FWPUBIP -l $REGION_NAME --sku "Standard"

 az network firewall ip-config create -g $RESOURCE_GROUP -f $FWNAME -n $FWIPCONFIG_NAME --public-ip-address $FWPUBIP --vnet-name $VNET_NAME

Azure automatically routes traffic between Azure subnets, virtual networks, and on-premises networks.

How to create a user-defined route table

You can create a UDR table with a hop to Azure Firewall.


export SUBID= <your Azure subscription ID>
export FWROUTE_TABLE_NAME=bdcaks-rt
export FWROUTE_NAME=bdcaksroute
export FWROUTE_NAME_INTERNET=bdcaksrouteinet

export FWPUBLIC_IP=$(az network public-ip show -g $RESOURCE_GROUP -n $FWPUBIP --query "ipAddress" -o tsv)
export FWPRIVATE_IP=$(az network firewall show -g $RESOURCE_GROUP -n $FWNAME --query "ipConfigurations[0].privateIpAddress" -o tsv)

# Create UDR and add a route for Azure Firewall

az network route-table create -g $RESOURCE_GROUP --name $FWROUTE_TABLE_NAME

az network route-table route create -g $RESOURCE_GROUP --name $FWROUTE_NAME --route-table-name $FWROUTE_TABLE_NAME --address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FWPRIVATE_IP --subscription $SUBID

az network route-table route create -g $RESOURCE_GROUP --name $FWROUTE_NAME_INTERNET --route-table-name $FWROUTE_TABLE_NAME --address-prefix $FWPUBLIC_IP/32 --next-hop-type Internet

How to set firewall rules

# Add FW Network Rules

az network firewall network-rule create -g $RESOURCE_GROUP -f $FWNAME --collection-name 'aksfwnr' -n 'apiudp' --protocols 'UDP' --source-addresses '*' --destination-addresses "AzureCloud.$REGION_NAME" --destination-ports 1194 --action allow --priority 100
az network firewall network-rule create -g $RESOURCE_GROUP -f $FWNAME --collection-name 'aksfwnr' -n 'apitcp' --protocols 'TCP' --source-addresses '*' --destination-addresses "AzureCloud.$REGION_NAME" --destination-ports 9000
az network firewall network-rule create -g $RESOURCE_GROUP -f $FWNAME --collection-name 'aksfwnr' -n 'time' --protocols 'UDP' --source-addresses '*' --destination-fqdns 'ntp.ubuntu.com' --destination-ports 123

# Add FW Application Rules

az network firewall application-rule create -g $RESOURCE_GROUP -f $FWNAME --collection-name 'aksfwar' -n 'fqdn' --source-addresses '*' --protocols 'http=80' 'https=443' --fqdn-tags "AzureKubernetesService" --action allow --priority 100

You can associate a UDR with an AKS cluster where you previously deployed a BDC, using the following command:

az network vnet subnet update -g $RESOURCE_GROUP --vnet-name $VNET_NAME --name $SUBNET_NAME --route-table $FWROUTE_TABLE_NAME

Create & configure the service principal (SP)

In this step, you need to create the service principal and assign permission to the virtual network.

See the following example:

# Create SP and Assign Permission to Virtual Network

az ad sp create-for-rbac -n "bdcaks-sp"

APPID=<your service principal ID >
PASSWORD=< your service principal password >
VNETID=$(az network vnet show -g $RESOURCE_GROUP --name $VNET_NAME --query id -o tsv)

# Assign SP Permission to VNET

az role assignment create --assignee $APPID --scope $VNETID --role "Network Contributor"


RTID=$(az network route-table show -g $RESOURCE_GROUP -n $FWROUTE_TABLE_NAME --query id -o tsv)
az role assignment create --assignee $APPID --scope $RTID --role "Network Contributor"

Create an AKS cluster

You can now create the AKS cluster with userDefinedRouting as outbound type.

az aks create \
    --resource-group $RESOURCE_GROUP \
    --location $REGION_NAME \
    --name $AKS_NAME \
    --load-balancer-sku standard \
    --outbound-type userDefinedRouting \
    --enable-private-cluster \
    --network-plugin azure \
    --vnet-subnet-id $SUBNET_ID \
    --docker-bridge-address 172.17.0.1/16 \
    --dns-service-ip 10.2.0.10 \
    --service-cidr 10.2.0.0/24 \
    --service-principal $APPID \
    --client-secret $PASSWORD \
    --node-vm-size Standard_D13_v2 \
    --node-count 2 \
    --generate-ssh-keys

Build a Big Data Cluster deployment profile

You can create a big data cluster with a custom profile:

azdata bdc config init --source aks-dev-test --target private-bdc-aks --force

Generate and configure a custom BDC deployment profile

azdata bdc config replace -c private-bdc-aks/control.json -j "$.spec.docker.imageTag=2019-CU6-ubuntu-16.04"
azdata bdc config replace -c private-bdc-aks/control.json -j "$.spec.storage.data.className=default"
azdata bdc config replace -c private-bdc-aks/control.json -j "$.spec.storage.logs.className=default"

azdata bdc config replace -c private-bdc-aks/control.json -j "$.spec.endpoints[0].serviceType=NodePort"
azdata bdc config replace -c private-bdc-aks/control.json -j "$.spec.endpoints[1].serviceType=NodePort"

azdata bdc config replace -c private-bdc-aks/bdc.json -j "$.spec.resources.master.spec.endpoints[0].serviceType=NodePort"
azdata bdc config replace -c private-bdc-aks/bdc.json -j "$.spec.resources.gateway.spec.endpoints[0].serviceType=NodePort"
azdata bdc config replace -c private-bdc-aks/bdc.json -j "$.spec.resources.appproxy.spec.endpoints[0].serviceType=NodePort"

Deploy a BDC in AKS private cluster

export AZDATA_USERNAME=<your bdcadmin username>
export AZDATA_PASSWORD=< your bdcadmin password>

azdata bdc create --config-profile private-bdc-aks --accept-eula yes

Can I use third-party firewalls to restrict egress traffic?

You can use third-party firewalls to restrict egress traffic with a deployed BDC and AKS private cluster. To view an example, visit Azure Marketplace firewalls. Third-party firewalls can be used in private deployment solutions with more compliant configurations. The firewall should provide the following network rules:

View all of the required outbound network rules and FQDNs for AKS clusters. This URL also includes all of the wildcard HTTP/HTTPS endpoints and dependencies. These can vary with your AKS cluster, based on a number of qualifiers, and your actual requirements.
Azure Global required network rules / FQDN/application rules mentioned here.
Optional recommended FQDN / application rules for AKS clusters mentioned here.

Please check how to manage a big data cluster in AKS private cluster and then the next step is to connect to a big data cluster.

See automation scripts for this scenario at SQL Server Samples repository on GitHub.

Restrict egress traffic from SQL Server 2019 Big Data Clusters in Azure Kubernetes Service (AKS) private cluster

How to restrict egress traffic with Azure firewall

Create the resource group and VNet

Create and set up Azure Firewall

How to create a user-defined route table

How to set firewall rules

Create & configure the service principal (SP)

Create an AKS cluster

Build a Big Data Cluster deployment profile

Generate and configure a custom BDC deployment profile

Deploy a BDC in AKS private cluster

Can I use third-party firewalls to restrict egress traffic?

Feedback

Feedback

Additional resources

Restrict egress traffic from SQL Server 2019 Big Data Clusters in Azure Kubernetes Service (AKS) private cluster

How to restrict egress traffic with Azure firewall

Create the resource group and VNet

Create and set up Azure Firewall

How to create a user-defined route table

How to set firewall rules

Create & configure the service principal (SP)

Create an AKS cluster

Build a Big Data Cluster deployment profile

Generate and configure a custom BDC deployment profile

Deploy a BDC in AKS private cluster

Can I use third-party firewalls to restrict egress traffic?

Related content

Feedback

Feedback

Additional resources