控制 Azure Kubernetes Service (AKS) 中叢集節點的連出流量Control egress traffic for cluster nodes in Azure Kubernetes Service (AKS)

本文提供必要的詳細資料,可讓您保護來自 Azure Kubernetes Service (AKS) 的輸出流量。This article provides the necessary details that allow you to secure outbound traffic from your Azure Kubernetes Service (AKS). 它包含基本 AKS 部署的叢集需求,以及選用附加元件和功能的其他需求。It contains the cluster requirements for a base AKS deployment, and additional requirements for optional addons and features. 如需如何使用 Azure 防火牆來設定這些需求的結尾,將會提供範例An example will be provided at the end on how to configure these requirements with Azure Firewall. 不過,您可以將此資訊套用至任何輸出限制方法或設備。However, you can apply this information to any outbound restriction method or appliance.

背景Background

AKS 叢集會部署在虛擬網路上。AKS clusters are deployed on a virtual network. 此網路可由 AKS) 所建立,或由使用者事先) 預先設定的自訂 (來管理 (。This network can be managed (created by AKS) or custom (pre-configured by the user beforehand). 無論是哪一種情況,叢集都有該虛擬網路外的服務 輸出 相依性 (服務沒有) 的輸入相依性。In either case, the cluster has outbound dependencies on services outside of that virtual network (the service has no inbound dependencies).

基於管理和操作目的,AKS 叢集中的節點需要存取特定連接埠和完整網域名稱 (FQDN)。For management and operational purposes, nodes in an AKS cluster need to access certain ports and fully qualified domain names (FQDNs). 節點需要這些端點才能與 API 伺服器通訊,或下載並安裝核心 Kubernetes 叢集元件和節點安全性更新。These endpoints are required for the nodes to communicate with the API server, or to download and install core Kubernetes cluster components and node security updates. 例如,叢集必須從 Microsoft Container Registry 提取基礎系統容器映射 (MCR) 。For example, the cluster needs to pull base system container images from Microsoft Container Registry (MCR).

AKS 輸出相依性幾乎完全以 Fqdn 定義,其後面沒有靜態位址。The AKS outbound dependencies are almost entirely defined with FQDNs, which don't have static addresses behind them. 缺少靜態位址表示網路安全性群組不能用來鎖定來自 AKS 叢集的輸出流量。The lack of static addresses means that Network Security Groups can't be used to lock down the outbound traffic from an AKS cluster.

根據預設,AKS 叢集具有不受限制的輸出 (連出) 網際網路存取。By default, AKS clusters have unrestricted outbound (egress) internet access. 此網路存取層級可讓您執行的節點和服務視需要存取外部資源。This level of network access allows nodes and services you run to access external resources as needed. 如果您想要限制連出流量,則必須能夠存取有限數量的連接埠和位址,才能維持狀況良好的叢集維護工作。If you wish to restrict egress traffic, a limited number of ports and addresses must be accessible to maintain healthy cluster maintenance tasks. 保護輸出位址最簡單的解決方案是使用防火牆裝置,該裝置可以根據功能變數名稱來控制輸出流量。The simplest solution to securing outbound addresses lies in use of a firewall device that can control outbound traffic based on domain names. 例如,Azure 防火牆可以根據目的地的 FQDN 限制輸出 HTTP 和 HTTPS 流量。Azure Firewall, for example, can restrict outbound HTTP and HTTPS traffic based on the FQDN of the destination. 您也可以設定慣用的防火牆和安全性規則,以允許這些必要的埠和位址。You can also configure your preferred firewall and security rules to allow these required ports and addresses.

重要

此文件僅涵蓋如何鎖定離開 AKS 子網路的流量。This document covers only how to lock down the traffic leaving the AKS subnet. 依預設,AKS 沒有輸入需求。AKS has no ingress requirements by default. 不支援使用網路安全性群組來封鎖 內部子網流量 (nsg) 和防火牆。Blocking internal subnet traffic using network security groups (NSGs) and firewalls is not supported. 若要控制和封鎖叢集中的流量,請使用 網路原則To control and block the traffic within the cluster, use Network Policies.

AKS 叢集所需的輸出網路規則和 FqdnRequired outbound network rules and FQDNs for AKS clusters

AKS 叢集需要下列網路和 FQDN/應用程式規則,如果您想要設定 Azure 防火牆以外的解決方案,您可以使用它們。The following network and FQDN/application rules are required for an AKS cluster, you can use them if you wish to configure a solution other than Azure Firewall.

  • 適用於非 HTTP/S 流量的 IP 地址相依性 (TCP 與 UDP 流量)IP Address dependencies are for non-HTTP/S traffic (both TCP and UDP traffic)
  • FQDN HTTP/HTTPS 端點可以放在您的防火牆裝置。FQDN HTTP/HTTPS endpoints can be placed in your firewall device.
  • 萬用字元 HTTP/HTTPS 端點是相依性,可能會因數個限定詞而不同于您的 AKS 叢集。Wildcard HTTP/HTTPS endpoints are dependencies that can vary with your AKS cluster based on a number of qualifiers.
  • AKS 會使用許可控制器將 FQDN 做為環境變數插入 kube 系統和閘道管理員系統下的所有部署,以確保節點和 API 伺服器之間的所有系統通訊都使用 API 伺服器 FQDN,而不是 API 伺服器 IP。AKS uses an admission controller to inject the FQDN as an environment variable to all deployments under kube-system and gatekeeper-system, that ensures all system communication between nodes and API server uses the API server FQDN and not the API server IP.
  • 如果您有需要與 API 伺服器通訊的應用程式或解決方案,則必須新增 額外 的網路規則,以允許 對 api 伺服器 IP 的埠443進行 TCP 通訊If you have an app or solution that needs to talk to the API server, you must add an additional network rule to allow TCP communication to port 443 of your API server's IP.
  • 在罕見的情況下,如果有維護作業,則您的 API 伺服器 IP 可能會變更。On rare occasions, if there's a maintenance operation your API server IP might change. 可變更 API 伺服器 IP 的預定維護作業,一律會事先進行通訊。Planned maintenance operations that can change the API server IP are always communicated in advance.

Azure 全球所需的網路規則Azure Global required network rules

必要的網路規則和 IP 位址相依性如下:The required network rules and IP address dependencies are:

目的地端點Destination Endpoint 通訊協定Protocol 連接埠Port 使用Use
*:1194
OrOr
ServiceTag - AzureCloud.<Region>:1194ServiceTag - AzureCloud.<Region>:1194
OrOr
區域 Cidr - RegionCIDRs:1194Regional CIDRs - RegionCIDRs:1194
OrOr
APIServerPublicIP:1194 (only known after cluster creation)APIServerPublicIP:1194 (only known after cluster creation)
UDPUDP 11941194 用於節點和控制平面之間的通道通訊安全通訊。For tunneled secure communication between the nodes and the control plane. 私人叢集不需要這項功能This is not required for private clusters
*:9000
OrOr
ServiceTag - AzureCloud.<Region>:9000ServiceTag - AzureCloud.<Region>:9000
OrOr
區域 Cidr - RegionCIDRs:9000Regional CIDRs - RegionCIDRs:9000
OrOr
APIServerPublicIP:9000 (only known after cluster creation)APIServerPublicIP:9000 (only known after cluster creation)
TCPTCP 90009000 用於節點和控制平面之間的通道通訊安全通訊。For tunneled secure communication between the nodes and the control plane. 私人叢集不需要這項功能This is not required for private clusters
*:123 或者, ntp.ubuntu.com:123 如果使用 Azure 防火牆網路規則,則 () *:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDPUDP 123123 網路時間通訊協定需要 (NTP) Linux 節點上的時間同步處理。Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers)CustomDNSIP:53 (if using custom DNS servers) UDPUDP 5353 如果您使用自訂的 DNS 伺服器,則必須確定叢集節點可存取它們。If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerPublicIP:443 (if running pods/deployments that access the API Server)APIServerPublicIP:443 (if running pods/deployments that access the API Server) TCPTCP 443443 如果執行存取 API 伺服器的 pod/部署,則為必要項,這些 pod/部署會使用 API IP。Required if running pods/deployments that access the API Server, those pods/deployments would use the API IP. 私人叢集不需要這項功能This is not required for private clusters

Azure 全域必要的 FQDN/應用程式規則Azure Global required FQDN / application rules

需要下列 FQDN/應用程式規則:The following FQDN / application rules are required:

目的地 FQDNDestination FQDN 連接埠Port 使用Use
*.hcp.<location>.azmk8s.io HTTPS:443 節點 < > API 伺服器通訊的必要元件。Required for Node <-> API server communication. 取代 <location> 為您的 AKS 叢集部署所在的區域。Replace <location> with the region where your AKS cluster is deployed.
mcr.microsoft.com HTTPS:443 存取 Microsoft Container Registry 中的映射所需 (MCR) 。Required to access images in Microsoft Container Registry (MCR). 此登錄包含第一方映射/圖表 (例如,coreDNS 等 ) 。This registry contains first-party images/charts (for example, coreDNS, etc.). 您必須要有這些映射,才能正確建立和運作叢集,包括調整和升級作業。These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
*.data.mcr.microsoft.com HTTPS:443 Azure 內容傳遞網路支援的 MCR 儲存體 (CDN) 所需。Required for MCR storage backed by the Azure content delivery network (CDN).
management.azure.com HTTPS:443 針對 Azure API 進行 Kubernetes 作業所需。Required for Kubernetes operations against the Azure API.
login.microsoftonline.com HTTPS:443 Azure Active Directory 驗證的必要。Required for Azure Active Directory authentication.
packages.microsoft.com HTTPS:443 此位址是用於快取 apt-get 作業的 Microsoft 套件存放庫。This address is the Microsoft packages repository used for cached apt-get operations. 範例套件包括 Moby、PowerShell 和 Azure CLI。Example packages include Moby, PowerShell, and Azure CLI.
acs-mirror.azureedge.net HTTPS:443 此位址適用于下載和安裝所需的二進位檔(例如 kubenet 和 Azure CNI)所需的存放庫。This address is for the repository required to download and install required binaries like kubenet and Azure CNI.

Azure 中國世紀需要網路規則Azure China 21Vianet required network rules

必要的網路規則和 IP 位址相依性如下:The required network rules and IP address dependencies are:

目的地端點Destination Endpoint 通訊協定Protocol 連接埠Port 使用Use
*:1194
OrOr
ServiceTag - AzureCloud.Region:1194ServiceTag - AzureCloud.Region:1194
OrOr
區域 Cidr - RegionCIDRs:1194Regional CIDRs - RegionCIDRs:1194
OrOr
APIServerPublicIP:1194 (only known after cluster creation)APIServerPublicIP:1194 (only known after cluster creation)
UDPUDP 11941194 用於節點和控制平面之間的通道通訊安全通訊。For tunneled secure communication between the nodes and the control plane.
*:9000
OrOr
ServiceTag - AzureCloud.<Region>:9000ServiceTag - AzureCloud.<Region>:9000
OrOr
區域 Cidr - RegionCIDRs:9000Regional CIDRs - RegionCIDRs:9000
OrOr
APIServerPublicIP:9000 (only known after cluster creation)APIServerPublicIP:9000 (only known after cluster creation)
TCPTCP 90009000 用於節點和控制平面之間的通道通訊安全通訊。For tunneled secure communication between the nodes and the control plane.
*:22
OrOr
ServiceTag - AzureCloud.<Region>:22ServiceTag - AzureCloud.<Region>:22
OrOr
區域 Cidr - RegionCIDRs:22Regional CIDRs - RegionCIDRs:22
OrOr
APIServerPublicIP:22 (only known after cluster creation)APIServerPublicIP:22 (only known after cluster creation)
TCPTCP 2222 用於節點和控制平面之間的通道通訊安全通訊。For tunneled secure communication between the nodes and the control plane.
*:123 或者, ntp.ubuntu.com:123 如果使用 Azure 防火牆網路規則,則 () *:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDPUDP 123123 網路時間通訊協定需要 (NTP) Linux 節點上的時間同步處理。Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers)CustomDNSIP:53 (if using custom DNS servers) UDPUDP 5353 如果您使用自訂的 DNS 伺服器,則必須確定叢集節點可存取它們。If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerPublicIP:443 (if running pods/deployments that access the API Server)APIServerPublicIP:443 (if running pods/deployments that access the API Server) TCPTCP 443443 如果執行存取 API 伺服器的 pod/部署,則為必要項,這些 pod/部署會使用 API IP。Required if running pods/deployments that access the API Server, those pod/deployments would use the API IP.

Azure 中國世紀需要 FQDN/應用程式規則Azure China 21Vianet required FQDN / application rules

需要下列 FQDN/應用程式規則:The following FQDN / application rules are required:

目的地 FQDNDestination FQDN 連接埠Port 使用Use
*.hcp.<location>.cx.prod.service.azk8s.cn HTTPS:443 節點 < > API 伺服器通訊的必要元件。Required for Node <-> API server communication. 取代 <location> 為您的 AKS 叢集部署所在的區域。Replace <location> with the region where your AKS cluster is deployed.
*.tun.<location>.cx.prod.service.azk8s.cn HTTPS:443 節點 < > API 伺服器通訊的必要元件。Required for Node <-> API server communication. 取代 <location> 為您的 AKS 叢集部署所在的區域。Replace <location> with the region where your AKS cluster is deployed.
mcr.microsoft.com HTTPS:443 存取 Microsoft Container Registry 中的映射所需 (MCR) 。Required to access images in Microsoft Container Registry (MCR). 此登錄包含第一方映射/圖表 (例如,coreDNS 等 ) 。This registry contains first-party images/charts (for example, coreDNS, etc.). 您必須要有這些映射,才能正確建立和運作叢集,包括調整和升級作業。These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
.data.mcr.microsoft.com HTTPS:443 Azure 內容傳遞網路支援的 MCR 儲存體 (CDN) 所需。Required for MCR storage backed by the Azure Content Delivery Network (CDN).
management.chinacloudapi.cn HTTPS:443 針對 Azure API 進行 Kubernetes 作業所需。Required for Kubernetes operations against the Azure API.
login.chinacloudapi.cn HTTPS:443 Azure Active Directory 驗證的必要。Required for Azure Active Directory authentication.
packages.microsoft.com HTTPS:443 此位址是用於快取 apt-get 作業的 Microsoft 套件存放庫。This address is the Microsoft packages repository used for cached apt-get operations. 範例套件包括 Moby、PowerShell 和 Azure CLI。Example packages include Moby, PowerShell, and Azure CLI.
*.azk8s.cn HTTPS:443 此位址適用于下載和安裝所需的二進位檔(例如 kubenet 和 Azure CNI)所需的存放庫。This address is for the repository required to download and install required binaries like kubenet and Azure CNI.

Azure 美國政府所需的網路規則Azure US Government required network rules

必要的網路規則和 IP 位址相依性如下:The required network rules and IP address dependencies are:

目的地端點Destination Endpoint 通訊協定Protocol 連接埠Port 使用Use
*:1194
OrOr
ServiceTag - AzureCloud.<Region>:1194ServiceTag - AzureCloud.<Region>:1194
OrOr
區域 Cidr - RegionCIDRs:1194Regional CIDRs - RegionCIDRs:1194
OrOr
APIServerPublicIP:1194 (only known after cluster creation)APIServerPublicIP:1194 (only known after cluster creation)
UDPUDP 11941194 用於節點和控制平面之間的通道通訊安全通訊。For tunneled secure communication between the nodes and the control plane.
*:9000
OrOr
ServiceTag - AzureCloud.<Region>:9000ServiceTag - AzureCloud.<Region>:9000
OrOr
區域 Cidr - RegionCIDRs:9000Regional CIDRs - RegionCIDRs:9000
OrOr
APIServerPublicIP:9000 (only known after cluster creation)APIServerPublicIP:9000 (only known after cluster creation)
TCPTCP 90009000 用於節點和控制平面之間的通道通訊安全通訊。For tunneled secure communication between the nodes and the control plane.
*:123 或者, ntp.ubuntu.com:123 如果使用 Azure 防火牆網路規則,則 () *:123 or ntp.ubuntu.com:123 (if using Azure Firewall network rules) UDPUDP 123123 網路時間通訊協定需要 (NTP) Linux 節點上的時間同步處理。Required for Network Time Protocol (NTP) time synchronization on Linux nodes.
CustomDNSIP:53 (if using custom DNS servers)CustomDNSIP:53 (if using custom DNS servers) UDPUDP 5353 如果您使用自訂的 DNS 伺服器,則必須確定叢集節點可存取它們。If you're using custom DNS servers, you must ensure they're accessible by the cluster nodes.
APIServerPublicIP:443 (if running pods/deployments that access the API Server)APIServerPublicIP:443 (if running pods/deployments that access the API Server) TCPTCP 443443 如果執行存取 API 伺服器的 pod/部署,則為必要項,這些 pod/部署會使用 API IP。Required if running pods/deployments that access the API Server, those pods/deployments would use the API IP.

Azure 美國政府需要 FQDN/應用程式規則Azure US Government required FQDN / application rules

需要下列 FQDN/應用程式規則:The following FQDN / application rules are required:

目的地 FQDNDestination FQDN 連接埠Port 使用Use
*.hcp.<location>.cx.aks.containerservice.azure.us HTTPS:443 節點 < > API 伺服器通訊的必要元件。Required for Node <-> API server communication. 取代 <location> 為您的 AKS 叢集部署所在的區域。Replace <location> with the region where your AKS cluster is deployed.
mcr.microsoft.com HTTPS:443 存取 Microsoft Container Registry 中的映射所需 (MCR) 。Required to access images in Microsoft Container Registry (MCR). 此登錄包含第一方映射/圖表 (例如,coreDNS 等 ) 。This registry contains first-party images/charts (for example, coreDNS, etc.). 您必須要有這些映射,才能正確建立和運作叢集,包括調整和升級作業。These images are required for the correct creation and functioning of the cluster, including scale and upgrade operations.
*.data.mcr.microsoft.com HTTPS:443 Azure 內容傳遞網路支援的 MCR 儲存體 (CDN) 所需。Required for MCR storage backed by the Azure content delivery network (CDN).
management.usgovcloudapi.net HTTPS:443 針對 Azure API 進行 Kubernetes 作業所需。Required for Kubernetes operations against the Azure API.
login.microsoftonline.us HTTPS:443 Azure Active Directory 驗證的必要。Required for Azure Active Directory authentication.
packages.microsoft.com HTTPS:443 此位址是用於快取 apt-get 作業的 Microsoft 套件存放庫。This address is the Microsoft packages repository used for cached apt-get operations. 範例套件包括 Moby、PowerShell 和 Azure CLI。Example packages include Moby, PowerShell, and Azure CLI.
acs-mirror.azureedge.net HTTPS:443 此位址適用於安裝必要的二進位檔 (例如 kubenet 和 Azure CNI) 所需的存放庫。This address is for the repository required to install required binaries like kubenet and Azure CNI.

下列 FQDN/應用程式規則是選擇性的,但建議用於 AKS 叢集:The following FQDN / application rules are optional but recommended for AKS clusters:

目的地 FQDNDestination FQDN 連接埠Port 使用Use
security.ubuntu.com, azure.archive.ubuntu.com, changelogs.ubuntu.comsecurity.ubuntu.com, azure.archive.ubuntu.com, changelogs.ubuntu.com HTTP:80 此位址可讓 Linux 叢集節點下載所需的安全性修補程式和更新。This address lets the Linux cluster nodes download the required security patches and updates.

如果您選擇封鎖/不允許這些 Fqdn,節點將只會在您進行 節點映射升級 或叢集 升級時收到作業系統更新。If you choose to block/not allow these FQDNs, the nodes will only receive OS updates when you do a node image upgrade or cluster upgrade.

已啟用 GPU 的 AKS 叢集GPU enabled AKS clusters

必要的 FQDN/應用程式規則Required FQDN / application rules

已啟用 GPU 之 AKS 叢集需要下列 FQDN/應用程式規則:The following FQDN / application rules are required for AKS clusters that have GPU enabled:

目的地 FQDNDestination FQDN 連接埠Port 使用Use
nvidia.github.io HTTPS:443 此位址是用來在以 GPU 為基礎的節點上進行正確的驅動程式安裝和運作。This address is used for correct driver installation and operation on GPU-based nodes.
us.download.nvidia.com HTTPS:443 此位址是用來在以 GPU 為基礎的節點上進行正確的驅動程式安裝和運作。This address is used for correct driver installation and operation on GPU-based nodes.
apt.dockerproject.org HTTPS:443 此位址是用來在以 GPU 為基礎的節點上進行正確的驅動程式安裝和運作。This address is used for correct driver installation and operation on GPU-based nodes.

以 Windows Server 為基礎的節點集區Windows Server based node pools

必要的 FQDN/應用程式規則Required FQDN / application rules

使用 Windows Server 型節點集區時,需要下列 FQDN/應用程式規則:The following FQDN / application rules are required for using Windows Server based node pools:

目的地 FQDNDestination FQDN 連接埠Port 使用Use
onegetcdn.azureedge.net, go.microsoft.com HTTPS:443 安裝與 Windows 相關的二進位檔To install windows-related binaries
*.mp.microsoft.com, www.msftconnecttest.com, ctldl.windowsupdate.com HTTP:80 安裝與 Windows 相關的二進位檔To install windows-related binaries

AKS 附加元件和整合AKS addons and integrations

適用於容器的 Azure 監視器Azure Monitor for containers

有兩個選項可讓您存取容器的 Azure 監視器,您可以允許 Azure 監視器 ServiceTag 提供所需 FQDN/應用程式規則的存取權。There are two options to provide access to Azure Monitor for containers, you may allow the Azure Monitor ServiceTag or provide access to the required FQDN/Application Rules.

必要的網路規則Required network rules

需要下列 FQDN/應用程式規則:The following FQDN / application rules are required:

目的地端點Destination Endpoint 通訊協定Protocol 連接埠Port 使用Use
ServiceTag - AzureMonitor:443ServiceTag - AzureMonitor:443 TCPTCP 443443 此端點是用來將計量資料和記錄傳送至 Azure 監視器和 Log Analytics。This endpoint is used to send metrics data and logs to Azure Monitor and Log Analytics.

必要的 FQDN/應用程式規則Required FQDN / application rules

已啟用適用於容器之 Azure 監視器的 AKS 叢集需要下列 FQDN/應用程式規則:The following FQDN / application rules are required for AKS clusters that have the Azure Monitor for containers enabled:

FQDNFQDN 連接埠Port 使用Use
dc.services.visualstudio.comdc.services.visualstudio.com HTTPS:443 此端點會用於使用 Azure 監視器的計量和監視遙測。This endpoint is used for metrics and monitoring telemetry using Azure Monitor.
*.ods.opinsights.azure.com*.ods.opinsights.azure.com HTTPS:443 Azure 監視器使用此端點來擷取 log analytics 資料。This endpoint is used by Azure Monitor for ingesting log analytics data.
*.oms.opinsights.azure.com*.oms.opinsights.azure.com HTTPS:443 Omsagent 會使用此端點來驗證 log analytics 服務。This endpoint is used by omsagent, which is used to authenticate the log analytics service.
*.monitoring.azure.com*.monitoring.azure.com HTTPS:443 此端點是用來將計量資料傳送至 Azure 監視器。This endpoint is used to send metrics data to Azure Monitor.

Azure Dev SpacesAzure Dev Spaces

更新您的防火牆或安全性設定,以允許下列所有 Fqdn 和 Azure Dev Spaces 基礎結構服務之間的網路流量。Update your firewall or security configuration to allow network traffic to and from the all of the below FQDNs and Azure Dev Spaces infrastructure services.

必要的網路規則Required network rules

目的地端點Destination Endpoint 通訊協定Protocol 連接埠Port 使用Use
ServiceTag - AzureDevSpacesServiceTag - AzureDevSpaces TCPTCP 443443 此端點是用來將計量資料和記錄傳送至 Azure 監視器和 Log Analytics。This endpoint is used to send metrics data and logs to Azure Monitor and Log Analytics.

必要的 FQDN/應用程式規則Required FQDN / application rules

已啟用 Azure Dev Spaces 的 AKS 叢集需要下列 FQDN/應用程式規則:The following FQDN / application rules are required for AKS clusters that have the Azure Dev Spaces enabled:

FQDNFQDN 連接埠Port 使用Use
cloudflare.docker.com HTTPS:443 此位址是用來提取 Linux Alpine 和其他 Azure Dev Spaces 映像This address is used to pull linux alpine and other Azure Dev Spaces images
gcr.io HTTPS:443 此位址是用來提取 helm/tiller 映像This address is used to pull helm/tiller images
storage.googleapis.com HTTPS:443 此位址是用來提取 helm/tiller 映像This address is used to pull helm/tiller images

Azure 原則Azure Policy

必要的 FQDN/應用程式規則Required FQDN / application rules

已啟用 Azure 原則的 AKS 叢集需要下列 FQDN/應用程式規則。The following FQDN / application rules are required for AKS clusters that have the Azure Policy enabled.

FQDNFQDN 連接埠Port 使用Use
data.policy.core.windows.net HTTPS:443 此位址可用來提取 Kubernetes 原則,並將叢集合規性狀態回報給原則服務。This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service.
store.policy.core.windows.net HTTPS:443 此位址用來提取內建原則的閘道管理員構件。This address is used to pull the Gatekeeper artifacts of built-in policies.
dc.services.visualstudio.com HTTPS:443 Azure 原則附加元件,其會將遙測資料傳送至 Application Insights 端點。Azure Policy add-on that sends telemetry data to applications insights endpoint.

Azure 中國世紀需要 FQDN/應用程式規則Azure China 21Vianet Required FQDN / application rules

已啟用 Azure 原則的 AKS 叢集需要下列 FQDN/應用程式規則。The following FQDN / application rules are required for AKS clusters that have the Azure Policy enabled.

FQDNFQDN 連接埠Port 使用Use
data.policy.azure.cn HTTPS:443 此位址可用來提取 Kubernetes 原則,並將叢集合規性狀態回報給原則服務。This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service.
store.policy.azure.cn HTTPS:443 此位址用來提取內建原則的閘道管理員構件。This address is used to pull the Gatekeeper artifacts of built-in policies.

Azure 美國政府需要 FQDN/應用程式規則Azure US Government Required FQDN / application rules

已啟用 Azure 原則的 AKS 叢集需要下列 FQDN/應用程式規則。The following FQDN / application rules are required for AKS clusters that have the Azure Policy enabled.

FQDNFQDN 連接埠Port 使用Use
data.policy.azure.us HTTPS:443 此位址可用來提取 Kubernetes 原則,並將叢集合規性狀態回報給原則服務。This address is used to pull the Kubernetes policies and to report cluster compliance status to policy service.
store.policy.azure.us HTTPS:443 此位址用來提取內建原則的閘道管理員構件。This address is used to pull the Gatekeeper artifacts of built-in policies.

使用 Azure 防火牆限制輸出流量Restrict egress traffic using Azure firewall

Azure 防火牆會提供 Azure Kubernetes Service (AzureKubernetesService) FQDN 標籤,以簡化此設定。Azure Firewall provides an Azure Kubernetes Service (AzureKubernetesService) FQDN Tag to simplify this configuration.

注意

FQDN 標籤包含上列所有 Fqdn,並會自動保持在最新狀態。The FQDN tag contains all the FQDNs listed above and is kept automatically up to date.

針對生產案例,建議在 Azure 防火牆上至少有20個前端 Ip,以避免發生 SNAT 埠耗盡問題。We recommend having a minimum of 20 Frontend IPs on the Azure Firewall for production scenarios to avoid incurring in SNAT port exhaustion issues.

以下是部署的範例架構:Below is an example architecture of the deployment:

已鎖定拓撲

  • 公用輸入強制流經防火牆篩選器Public Ingress is forced to flow through firewall filters
    • AKS 代理程式節點會獨立于專用子網中。AKS agent nodes are isolated in a dedicated subnet.
    • Azure 防火牆 會部署在它自己的子網中。Azure Firewall is deployed in its own subnet.
    • DNAT 規則會將 FW 公用 IP 轉譯為 LB 前端 IP。A DNAT rule translates the FW public IP into the LB frontend IP.
  • 輸出要求會使用使用者定義的路由,從代理程式節點開始至 Azure 防火牆內部 IPOutbound requests start from agent nodes to the Azure Firewall internal IP using a user-defined route
    • 來自 AKS 代理程式節點的要求會遵循已放置於部署 AKS 叢集之子網路上的 UDR。Requests from AKS agent nodes follow a UDR that has been placed on the subnet the AKS cluster was deployed into.
    • Azure 防火牆會從公用 IP 前端輸出虛擬網路Azure Firewall egresses out of the virtual network from a public IP frontend
    • 存取公用網際網路或其他 Azure 服務,會流入和流出防火牆前端 IP 位址Access to the public internet or other Azure services flows to and from the firewall frontend IP address
    • (選擇性) AKS 控制平面的存取權受 API 伺服器授權的 ip 範圍所保護,其中包含防火牆公用前端 ip 位址。Optionally, access to the AKS control plane is protected by API server Authorized IP ranges, which includes the firewall public frontend IP address.
  • 內部流量Internal Traffic

下列步驟會使用 Azure 防火牆的 FQDN 標籤 AzureKubernetesService 來限制來自 AKS 叢集的輸出流量,並提供如何透過防火牆設定公用輸入流量的範例。The below steps make use of Azure Firewall's AzureKubernetesService FQDN tag to restrict the outbound traffic from the AKS cluster and provide an example how to configure public inbound traffic via the firewall.

透過環境變數設定組態Set configuration via environment variables

定義一組要在資源建立時使用的環境變數。Define a set of environment variables to be used in resource creations.

PREFIX="aks-egress"
RG="${PREFIX}-rg"
LOC="eastus"
PLUGIN=azure
AKSNAME="${PREFIX}"
VNET_NAME="${PREFIX}-vnet"
AKSSUBNET_NAME="aks-subnet"
# DO NOT CHANGE FWSUBNET_NAME - This is currently a requirement for Azure Firewall.
FWSUBNET_NAME="AzureFirewallSubnet"
FWNAME="${PREFIX}-fw"
FWPUBLICIP_NAME="${PREFIX}-fwpublicip"
FWIPCONFIG_NAME="${PREFIX}-fwconfig"
FWROUTE_TABLE_NAME="${PREFIX}-fwrt"
FWROUTE_NAME="${PREFIX}-fwrn"
FWROUTE_NAME_INTERNET="${PREFIX}-fwinternet"

建立有多個子網路的虛擬網路Create a virtual network with multiple subnets

布建具有兩個不同子網的虛擬網路,一個用於叢集,一個用於防火牆。Provision a virtual network with two separate subnets, one for the cluster, one for the firewall. (選擇性)您也可以建立一個用於內部服務輸入。Optionally you could also create one for internal service ingress.

空的網路拓撲

建立資源群組以保存所有資源。Create a resource group to hold all of the resources.

# Create Resource Group

az group create --name $RG --location $LOC

建立具有兩個子網的虛擬網路,以裝載 AKS 叢集和 Azure 防火牆。Create a virtual network with two subnets to host the AKS cluster and the Azure Firewall. 每個都將有自己的子網路。Each will have their own subnet. 讓我們從 AKS 網路開始。Let's start with the AKS network.

# Dedicated virtual network with AKS subnet

az network vnet create \
    --resource-group $RG \
    --name $VNET_NAME \
    --location $LOC \
    --address-prefixes 10.42.0.0/16 \
    --subnet-name $AKSSUBNET_NAME \
    --subnet-prefix 10.42.1.0/24

# Dedicated subnet for Azure Firewall (Firewall name cannot be changed)

az network vnet subnet create \
    --resource-group $RG \
    --vnet-name $VNET_NAME \
    --name $FWSUBNET_NAME \
    --address-prefix 10.42.2.0/24

使用 UDR 建立和設定 Azure 防火牆Create and set up an Azure Firewall with a UDR

您必須設定 Azure 防火牆的連入與連出規則。Azure Firewall inbound and outbound rules must be configured. 防火牆的主要目的是要讓組織將細微的輸入和輸出流量規則設定入和移出 AKS 叢集。The main purpose of the firewall is to enable organizations to configure granular ingress and egress traffic rules into and out of the AKS Cluster.

防火牆和 UDR

重要

如果您的叢集或應用程式所建立的大量輸出連線會導向至相同或較小的目的地子集,您可能需要更多的防火牆前端 ip,以避免提高每個前端 IP 的埠。If your cluster or application creates a large number of outbound connections directed to the same or small subset of destinations, you might require more firewall frontend IPs to avoid maxing out the ports per frontend IP. 如需有關如何建立具有多個 Ip 的 Azure 防火牆的詳細資訊,請參閱 這裡For more information on how to create an Azure firewall with multiple IPs, see here

建立將用來作為 Azure 防火牆前端位址的標準 SKU 公用 IP 資源。Create a standard SKU public IP resource that will be used as the Azure Firewall frontend address.

az network public-ip create -g $RG -n $FWPUBLICIP_NAME -l $LOC --sku "Standard"

註冊預覽 CLI 延伸模組以建立 Azure 防火牆。Register the preview cli-extension to create an Azure Firewall.

# Install Azure Firewall preview CLI extension

az extension add --name azure-firewall

# Deploy Azure Firewall

az network firewall create -g $RG -n $FWNAME -l $LOC --enable-dns-proxy true

現在可以將先前建立的 IP 位址指派給防火牆前端。The IP address created earlier can now be assigned to the firewall frontend.

注意

將公用 IP 位址設定為 Azure 防火牆可能需要幾分鐘的時間。Set up of the public IP address to the Azure Firewall may take a few minutes. 若要在網路規則上利用 FQDN,我們必須啟用 DNS proxy,當啟用時,防火牆會接聽埠53,並將 DNS 要求轉寄至上述指定的 DNS 伺服器。To leverage FQDN on network rules we need DNS proxy enabled, when enabled the firewall will listen on port 53 and will forward DNS requests to the DNS server specified above. 這可讓防火牆自動轉譯該 FQDN。This will allow the firewall to translate that FQDN automatically.

# Configure Firewall IP Config

az network firewall ip-config create -g $RG -f $FWNAME -n $FWIPCONFIG_NAME --public-ip-address $FWPUBLICIP_NAME --vnet-name $VNET_NAME

當上一個命令成功之後,儲存防火牆前端 IP 位址,以供稍後設定。When the previous command has succeeded, save the firewall frontend IP address for configuration later.

# Capture Firewall IP Address for Later Use

FWPUBLIC_IP=$(az network public-ip show -g $RG -n $FWPUBLICIP_NAME --query "ipAddress" -o tsv)
FWPRIVATE_IP=$(az network firewall show -g $RG -n $FWNAME --query "ipConfigurations[0].privateIpAddress" -o tsv)

注意

如果您對 AKS API 伺服器使用具有 授權 ip 位址範圍的安全存取,則需要將防火牆公用 ip 新增至授權的 ip 範圍。If you use secure access to the AKS API server with authorized IP address ranges, you need to add the firewall public IP into the authorized IP range.

在 Azure 防火牆中建立具有躍點的 UDRCreate a UDR with a hop to Azure Firewall

Azure 會自動路由傳送 Azure 子網路、虛擬網路及內部部署網路之間的流量。Azure automatically routes traffic between Azure subnets, virtual networks, and on-premises networks. 如果您想要變更任何 Azure 的預設路由,您可藉由建立路由表來執行此動作。If you want to change any of Azure's default routing, you do so by creating a route table.

建立要與指定子網路相關聯的空路由表。Create an empty route table to be associated with a given subnet. 路由表會將下一個躍點定義為上面建立的 Azure 防火牆。The route table will define the next hop as the Azure Firewall created above. 每個子網路可以有零個或一個與其相關聯的路由表。Each subnet can have zero or one route table associated to it.

# Create UDR and add a route for Azure Firewall

az network route-table create -g $RG -l $LOC --name $FWROUTE_TABLE_NAME
az network route-table route create -g $RG --name $FWROUTE_NAME --route-table-name $FWROUTE_TABLE_NAME --address-prefix 0.0.0.0/0 --next-hop-type VirtualAppliance --next-hop-ip-address $FWPRIVATE_IP --subscription $SUBID
az network route-table route create -g $RG --name $FWROUTE_NAME_INTERNET --route-table-name $FWROUTE_TABLE_NAME --address-prefix $FWPUBLIC_IP/32 --next-hop-type Internet

請參閱虛擬網路路由表文件,以了解如何覆寫 Azure 的預設系統路由,或將其他路由新增至子網路的路由表。See virtual network route table documentation about how you can override Azure's default system routes or add additional routes to a subnet's route table.

新增防火牆規則Adding firewall rules

以下是您可以用來在防火牆上設定的三個網路規則,您可能需要根據您的部署來調整這些規則。Below are three network rules you can use to configure on your firewall, you may need to adapt these rules based on your deployment. 第一個規則可讓您透過 TCP 存取埠9000。The first rule allows access to port 9000 via TCP. 第二個規則允許透過 UDP 存取埠1194和 123 (如果您要部署到 Azure 中國的世紀,您可能需要 更多) 。The second rule allows access to port 1194 and 123 via UDP (if you're deploying to Azure China 21Vianet, you might require more). 這兩個規則只會允許目的地為 Azure 區域 CIDR (在此案例中為美國東部)的流量。Both these rules will only allow traffic destined to the Azure Region CIDR that we're using, in this case East US. 最後,我們會透過 UDP 新增第三個網路規則,以開啟埠123至 ntp.ubuntu.com FQDN (新增 FQDN 作為網路規則是 Azure 防火牆的其中一個特定功能,而且您必須在使用自己的選項時加以調整) 。Finally, we'll add a third network rule opening port 123 to ntp.ubuntu.com FQDN via UDP (adding an FQDN as a network rule is one of the specific features of Azure Firewall, and you'll need to adapt it when using your own options).

設定網路規則之後,我們也會使用來新增應用程式規則, AzureKubernetesService 其中涵蓋可透過 TCP 埠443和埠80存取的所有必要 fqdn。After setting the network rules, we'll also add an application rule using the AzureKubernetesService that covers all needed FQDNs accessible through TCP port 443 and port 80.

# Add FW Network Rules

az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'apiudp' --protocols 'UDP' --source-addresses '*' --destination-addresses "AzureCloud.$LOC" --destination-ports 1194 --action allow --priority 100
az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'apitcp' --protocols 'TCP' --source-addresses '*' --destination-addresses "AzureCloud.$LOC" --destination-ports 9000
az network firewall network-rule create -g $RG -f $FWNAME --collection-name 'aksfwnr' -n 'time' --protocols 'UDP' --source-addresses '*' --destination-fqdns 'ntp.ubuntu.com' --destination-ports 123

# Add FW Application Rules

az network firewall application-rule create -g $RG -f $FWNAME --collection-name 'aksfwar' -n 'fqdn' --source-addresses '*' --protocols 'http=80' 'https=443' --fqdn-tags "AzureKubernetesService" --action allow --priority 100

若要深入了解 Azure 防火牆服務,請參閱 Azure 防火牆文件See Azure Firewall documentation to learn more about the Azure Firewall service.

將路由表關聯至 AKSAssociate the route table to AKS

若要將叢集與防火牆建立關聯,叢集子網路的專用子網路必須參考上面建立的路由表。To associate the cluster with the firewall, the dedicated subnet for the cluster's subnet must reference the route table created above. 藉由發出命令給保存叢集和防火牆的虛擬網路,來更新叢集子網路的路由表,即可完成關聯。Association can be done by issuing a command to the virtual network holding both the cluster and firewall to update the route table of the cluster's subnet.

# Associate route table with next hop to Firewall to the AKS subnet

az network vnet subnet update -g $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --route-table $FWROUTE_TABLE_NAME

將具有 UDR 連出類型的 AKS 部署到現有網路Deploy AKS with outbound type of UDR to the existing network

現在 AKS 叢集可以部署到現有的虛擬網路中。Now an AKS cluster can be deployed into the existing virtual network. 我們也會使用輸出類型 userDefinedRouting ,這項功能可確保任何輸出流量都會透過防火牆強制執行,而且預設不會有其他輸出路徑 (預設會使用 Load Balancer 輸出類型) 。We'll also use outbound type userDefinedRouting, this feature ensures any outbound traffic will be forced through the firewall and no other egress paths will exist (by default the Load Balancer outbound type could be used).

AKS 部署

建立可存取現有虛擬網路內佈建的服務主體Create a service principal with access to provision inside the existing virtual network

AKS 會使用叢集身分識別 (受控識別或服務主體) 來建立叢集資源。A cluster identity (managed identity or service principal) is used by AKS to create cluster resources. 在建立期間傳遞的服務主體可用來建立基礎 AKS 資源,例如 AKS 所使用的儲存體資源、Ip 和負載平衡器 (您也可以改為使用 受控識別) 。A service principal that is passed at create time is used to create underlying AKS resources such as Storage resources, IPs, and Load Balancers used by AKS (you may also use a managed identity instead). 如果未授與下列適當的許可權,您將無法布建 AKS 叢集。If not granted the appropriate permissions below, you won't be able to provision the AKS Cluster.

# Create SP and Assign Permission to Virtual Network

az ad sp create-for-rbac -n "${PREFIX}sp" --skip-assignment

現在,以先前命令輸出自動產生的服務主體應用程式識別碼和服務主體密碼取代 APPIDPASSWORDNow replace the APPID and PASSWORD below with the service principal appid and service principal password autogenerated by the previous command output. 我們將參考 VNET 資源識別碼,以將許可權授與服務主體,讓 AKS 能夠將資源部署到其中。We'll reference the VNET resource ID to grant the permissions to the service principal so AKS can deploy resources into it.

APPID="<SERVICE_PRINCIPAL_APPID_GOES_HERE>"
PASSWORD="<SERVICEPRINCIPAL_PASSWORD_GOES_HERE>"
VNETID=$(az network vnet show -g $RG --name $VNET_NAME --query id -o tsv)

# Assign SP Permission to VNET

az role assignment create --assignee $APPID --scope $VNETID --role "Network Contributor"

您可以在 檢查所需的詳細許可權。You can check the detailed permissions that are required here.

注意

如果您使用的是 kubenet 網路外掛程式,您必須將 AKS 服務主體或受控識別許可權授與預先建立的路由表,因為 kubenet 需要路由表來新增資源清單路由規則。If you're using the kubenet network plugin, you'll need to give the AKS service principal or managed identity permissions to the pre-created route table, since kubenet requires a route table to add neccesary routing rules.

RTID=$(az network route-table show -g $RG -n $FWROUTE_TABLE_NAME --query id -o tsv)
az role assignment create --assignee $APPID --scope $RTID --role "Network Contributor"

部署 AKSDeploy AKS

最後,AKS 叢集可以部署到我們專用於叢集的現有子網中。Finally, the AKS cluster can be deployed into the existing subnet we've dedicated for the cluster. 要部署到的目標子網路是使用環境變數 $SUBNETID 來定義。The target subnet to be deployed into is defined with the environment variable, $SUBNETID. 我們並未在先前的步驟中定義 $SUBNETID 變數。We didn't define the $SUBNETID variable in the previous steps. 若要設定子網路識別碼的值,您可以使用下列命令:To set the value for the subnet ID, you can use the following command:

SUBNETID=$(az network vnet subnet show -g $RG --vnet-name $VNET_NAME --name $AKSSUBNET_NAME --query id -o tsv)

您將定義輸出類型,以使用已存在於子網的 UDR。You'll define the outbound type to use the UDR that already exists on the subnet. 此設定可讓 AKS 略過負載平衡器的設定和 IP 布建。This configuration will enable AKS to skip the setup and IP provisioning for the load balancer.

重要

如需輸出類型 UDR (包括限制)的詳細資訊,請參閱 輸出輸出類型 UDRFor more information on outbound type UDR including limitations, see egress outbound type UDR.

提示

您可以將其他功能新增至叢集部署,例如 私人叢集中Additional features can be added to the cluster deployment such as Private Cluster.

您可以新增 api 伺服器授權 IP 範圍 的 AKS 功能,以將 api 伺服器存取限制為僅限防火牆的公用端點。The AKS feature for API server authorized IP ranges can be added to limit API server access to only the firewall's public endpoint. [授權的 IP 範圍] 功能在圖表中表示為選擇性。The authorized IP ranges feature is denoted in the diagram as optional. 啟用授權的 IP 範圍功能以限制 API 伺服器存取時,您的開發人員工具必須使用來自防火牆虛擬網路的 Jumpbox,或者您必須將所有開發人員端點新增至授權 IP 範圍。When enabling the authorized IP range feature to limit API server access, your developer tools must use a jumpbox from the firewall's virtual network or you must add all developer endpoints to the authorized IP range.

az aks create -g $RG -n $AKSNAME -l $LOC \
  --node-count 3 --generate-ssh-keys \
  --network-plugin $PLUGIN \
  --outbound-type userDefinedRouting \
  --service-cidr 10.41.0.0/16 \
  --dns-service-ip 10.41.0.10 \
  --docker-bridge-address 172.17.0.1/16 \
  --vnet-subnet-id $SUBNETID \
  --service-principal $APPID \
  --client-secret $PASSWORD \
  --api-server-authorized-ip-ranges $FWPUBLIC_IP

讓開發人員能夠存取 API 伺服器Enable developer access to the API server

如果您在上一個步驟中使用叢集的授權 IP 範圍,您必須將開發人員工具的 IP 位址新增至已核准 IP 範圍的 AKS 叢集清單,才能從該處存取 API 伺服器。If you used authorized IP ranges for the cluster on the previous step, you must add your developer tooling IP addresses to the AKS cluster list of approved IP ranges in order to access the API server from there. 另一個選項是在防火牆虛擬網路中的個別子網路內,使用所需的工具來設定 Jumpbox。Another option is to configure a jumpbox with the needed tooling inside a separate subnet in the Firewall's virtual network.

使用下列命令,在核准的範圍內新增另一個 IP 位址Add another IP address to the approved ranges with the following command

# Retrieve your IP address
CURRENT_IP=$(dig @resolver1.opendns.com ANY myip.opendns.com +short)

# Add to AKS approved list
az aks update -g $RG -n $AKSNAME --api-server-authorized-ip-ranges $CURRENT_IP/32

使用 [az aks get-認證] [az-aks] 命令,將設定 kubectl 為連線至新建立的 Kubernetes 叢集。Use the [az aks get-credentials][az-aks-get-credentials] command to configure kubectl to connect to your newly created Kubernetes cluster.

az aks get-credentials -g $RG -n $AKSNAME

部署公用服務Deploy a public service

您現在可以開始公開服務,並將應用程式部署到此叢集。You can now start exposing services and deploying applications to this cluster. 在此範例中,我們會公開公用服務,但您也可以選擇透過 內部負載平衡器公開內部服務。In this example, we'll expose a public service, but you may also choose to expose an internal service via internal load balancer.

公用服務 DNAT

將下列 yaml 複製到名為 example.yaml 的檔案,以部署 Azure 投票 App 應用程式。Deploy the Azure voting app application by copying the yaml below to a file named example.yaml.

# voting-storage-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: voting-storage
spec:
  replicas: 1
  selector:
    matchLabels:
      app: voting-storage
  template:
    metadata:
      labels:
        app: voting-storage
    spec:
      containers:
      - name: voting-storage
        image: mcr.microsoft.com/aks/samples/voting/storage:2.0
        args: ["--ignore-db-dir=lost+found"]
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 250m
            memory: 256Mi
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_ROOT_PASSWORD
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_USER
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_PASSWORD
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_DATABASE
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
---
# voting-storage-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: voting-storage-secret
type: Opaque
data:
  MYSQL_USER: ZGJ1c2Vy
  MYSQL_PASSWORD: UGFzc3dvcmQxMg==
  MYSQL_DATABASE: YXp1cmV2b3Rl
  MYSQL_ROOT_PASSWORD: UGFzc3dvcmQxMg==
---
# voting-storage-pv-claim.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
# voting-storage-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: voting-storage
  labels: 
    app: voting-storage
spec:
  ports:
  - port: 3306
    name: mysql
  selector:
    app: voting-storage
---
# voting-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: voting-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: voting-app
  template:
    metadata:
      labels:
        app: voting-app
    spec:
      containers:
      - name: voting-app
        image: mcr.microsoft.com/aks/samples/voting/app:2.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: MYSQL_HOST
          value: "voting-storage"
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_USER
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_PASSWORD
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_DATABASE
        - name: ANALYTICS_HOST
          value: "voting-analytics"
---
# voting-app-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: voting-app
  labels: 
    app: voting-app
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
    name: http
  selector:
    app: voting-app
---
# voting-analytics-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: voting-analytics
spec:
  replicas: 1
  selector:
    matchLabels:
      app: voting-analytics
      version: "2.0"
  template:
    metadata:
      labels:
        app: voting-analytics
        version: "2.0"
    spec:
      containers:
      - name: voting-analytics
        image: mcr.microsoft.com/aks/samples/voting/analytics:2.0
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
          name: http
        env:
        - name: MYSQL_HOST
          value: "voting-storage"
        - name: MYSQL_USER
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_USER
        - name: MYSQL_PASSWORD
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_PASSWORD
        - name: MYSQL_DATABASE
          valueFrom:
            secretKeyRef:
              name: voting-storage-secret
              key: MYSQL_DATABASE
---
# voting-analytics-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: voting-analytics
  labels: 
    app: voting-analytics
spec:
  ports:
  - port: 8080
    name: http
  selector:
    app: voting-analytics

執行下列動作來部署服務:Deploy the service by running:

kubectl apply -f example.yaml

將 DNAT 規則新增至 Azure 防火牆Add a DNAT rule to Azure Firewall

重要

當您使用 Azure 防火牆來限制連出流量,並建立使用者定義路由 (UDR) 來強制執行所有連出流量時,請務必在防火牆中建立適當的 DNAT 規則,以正確地允許連入流量。When you use Azure Firewall to restrict egress traffic and create a user-defined route (UDR) to force all egress traffic, make sure you create an appropriate DNAT rule in Firewall to correctly allow ingress traffic. 將 Azure 防火牆搭配 UDR 使用,會因為非對稱式路由而中斷連入設定。Using Azure Firewall with a UDR breaks the ingress setup due to asymmetric routing. (如果 AKS 子網路的預設路由會前往防火牆私人 IP 位址,但是您使用的是公用負載平衡器 - 類型為 LoadBalancer 的連入或 Kubernetes 服務,則會發生此問題)。(The issue occurs if the AKS subnet has a default route that goes to the firewall's private IP address, but you're using a public load balancer - ingress or Kubernetes service of type: LoadBalancer). 在此情況下,系統會透過傳入負載平衡器流量的公用 IP 位址接收它,但傳回路徑則會通過防火牆的私人 IP 位址。In this case, the incoming load balancer traffic is received via its public IP address, but the return path goes through the firewall's private IP address. 由於防火牆是具狀態,其會捨棄傳回封包,因為防火牆並不知道已建立的工作階段。Because the firewall is stateful, it drops the returning packet because the firewall isn't aware of an established session. 若要了解如何整合 Azure 防火牆與您的連入或服務負載平衡器,請參閱整合 Azure 防火牆與 Azure Standard Load Balancer (部分機器翻譯)。To learn how to integrate Azure Firewall with your ingress or service load balancer, see Integrate Azure Firewall with Azure Standard Load Balancer.

若要設定連入連線能力,必須將 DNAT 規則寫入至 Azure 防火牆。To configure inbound connectivity, a DNAT rule must be written to the Azure Firewall. 若要測試對叢集的連線能力,會針對防火牆前端公用 IP 位址定義規則,以路由傳送至內部服務所公開的內部 IP。To test connectivity to your cluster, a rule is defined for the firewall frontend public IP address to route to the internal IP exposed by the internal service.

目的地位址可以自訂,因為它是防火牆上要存取的埠。The destination address can be customized as it's the port on the firewall to be accessed. 轉譯的位址必須是內部負載平衡器的 IP 位址。The translated address must be the IP address of the internal load balancer. 轉譯的連接埠必須是針對您 Kubernetes 服務公開的連接埠。The translated port must be the exposed port for your Kubernetes service.

您必須指定指派給 Kubernetes 服務所建立之負載平衡器的內部 IP 位址。You'll need to specify the internal IP address assigned to the load balancer created by the Kubernetes service. 執行下列命令來擷取位址:Retrieve the address by running:

kubectl get services

所需的 IP 位址將列在 EXTERNAL-IP (外部 IP) 欄中,如下所示。The IP address needed will be listed in the EXTERNAL-IP column, similar to the following.

NAME               TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
kubernetes         ClusterIP      10.41.0.1       <none>        443/TCP        10h
voting-analytics   ClusterIP      10.41.88.129    <none>        8080/TCP       9m
voting-app         LoadBalancer   10.41.185.82    20.39.18.6    80:32718/TCP   9m
voting-storage     ClusterIP      10.41.221.201   <none>        3306/TCP       9m

藉由執行下列動作來取得服務 IP:Get the service IP by running:

SERVICE_IP=$(kubectl get svc voting-app -o jsonpath='{.status.loadBalancer.ingress[*].ip}')

藉由執行下列動作來新增 NAT 規則:Add the NAT rule by running:

az network firewall nat-rule create --collection-name exampleset --destination-addresses $FWPUBLIC_IP --destination-ports 80 --firewall-name $FWNAME --name inboundrule --protocols Any --resource-group $RG --source-addresses '*' --translated-port 80 --action Dnat --priority 100 --translated-address $SERVICE_IP

驗證連線能力Validate connectivity

在瀏覽器中瀏覽至 Azure 防火牆前端 IP 位址,以驗證連線能力。Navigate to the Azure Firewall frontend IP address in a browser to validate connectivity.

您應該會看到 AKS 投票應用程式。You should see the AKS voting app. 在此範例中,防火牆公用 IP 為 52.253.228.132In this example, the Firewall public IP was 52.253.228.132.

螢幕擷取畫面顯示具有貓、狗和重設等按鈕的 K S 投票應用程式。

清除資源Clean up resources

若要清理 Azure 資源,請刪除 AKS 資源群組。To clean up Azure resources, delete the AKS resource group.

az group delete -g $RG

下一步Next steps

在本文中,您已瞭解當您想要限制叢集的輸出流量時,所允許的埠和位址。In this article, you learned what ports and addresses to allow if you want to restrict egress traffic for the cluster. 您也已瞭解如何使用 Azure 防火牆來保護您的輸出流量。You also saw how to secure your outbound traffic using Azure Firewall.

如有需要,您可以將上述步驟一般化,以將流量轉送到您偏好的輸出解決方案,請遵循 輸出類型 userDefinedRouteIf needed, you can generalize the steps above to forward the traffic to your preferred egress solution, following the Outbound Type userDefinedRoute documentation.

如果您想要限制 pod 之間的通訊方式,以及叢集中的 East-West 流量限制,請參閱 使用 AKS 中的網路原則來保護 pod 之間的流量If you want to restrict how pods communicate between themselves and East-West traffic restrictions within cluster see Secure traffic between pods using network policies in AKS.