您现在访问的是微软AZURE全球版技术文档网站,若需要访问由世纪互联运营的MICROSOFT AZURE中国区技术文档网站,请访问 https://docs.azure.cn.

使用 REST 代理与 Azure HDInsight 中的 Apache Kafka 群集交互Interact with Apache Kafka clusters in Azure HDInsight using a REST proxy

使用 Kafka REST 代理可以通过基于 HTTP 的 REST API 来与 Kafka 群集交互。Kafka REST Proxy enables you to interact with your Kafka cluster via a REST API over HTTP. 此操作表示你的 Kafka 客户端可位于虚拟网络之外。This action means that your Kafka clients can be outside of your virtual network. 客户端可以对 Kafka 群集进行简单的 HTTP 调用,而不必依赖 Kafka 库。Clients can make simple HTTP calls to the Kafka cluster, instead of relying on Kafka libraries. 本文将演示如何创建启用了 REST 代理的 Kafka 群集。This article will show you how to create a REST proxy enabled Kafka cluster. 另外还提供了一个示例代码,演示如何调用 REST 代理。Also provides a sample code that shows how to make calls to REST proxy.

REST API 参考REST API reference

有关 Kafka REST API 支持的操作,请参阅 HDInsight Kafka REST 代理 API 参考For operations supported by the Kafka REST API, see HDInsight Kafka REST Proxy API Reference.

背景Background

Kafka REST 代理设计

有关 API 支持的操作的完整规范,请参阅 Apache Kafka REST 代理 APIFor the full specification of operations supported by the API, see Apache Kafka REST Proxy API.

REST 代理端点REST Proxy endpoint

使用 REST 代理创建 HDInsight Kafka 群集会为群集创建新的公共终结点,你可以在 Azure 门户的 HDInsight 群集“属性”中找到该终结点****。Creating an HDInsight Kafka cluster with REST proxy creates a new public endpoint for your cluster, which you can find in your HDInsight cluster Properties on the Azure portal.

安全性Security

使用 Azure Active Directory 安全组来管理对 Kafka REST 代理的访问。Access to the Kafka REST proxy is managed with Azure Active Directory security groups. 创建 Kafka 群集时,请为 Azure AD 安全组提供 REST 终结点访问权限。When creating the Kafka cluster, provide the Azure AD security group with REST endpoint access. 需要访问 REST 代理的 Kafka 客户端应由组所有者注册到此组。Kafka clients that need access to the REST proxy should be registered to this group by the group owner. 组所有者可通过门户或 PowerShell 注册。The group owner can register via the Portal or via PowerShell.

对于 REST 代理终结点请求,客户端应用程序应获取 OAuth 令牌。For REST proxy endpoint requests, client applications should get an OAuth token. 令牌用于验证安全组成员身份。The token is used to verify security group membership. 查找下面的客户端应用程序示例,其中演示了如何获取 OAuth 令牌。Find a Client application sample below that shows how to get an OAuth token. 客户端应用程序会在 HTTP 请求中将 OAuth 令牌传递给 REST 代理。The client application passes the OAuth token in the HTTP request to the REST proxy.

备注

请参阅使用 Azure Active Directory 组管理应用和资源访问来详细了解 AAD 安全组。See Manage app and resource access using Azure Active Directory groups, to learn more about AAD security groups. 有关 OAuth 令牌工作原理的详细信息,请参阅使用 OAuth 2.0 代码授权流来授权访问 Azure Active Directory Web 应用程序For more information on how OAuth tokens work, see Authorize access to Azure Active Directory web applications using the OAuth 2.0 code grant flow.

包含网络安全组的 Kafka REST 代理Kafka REST proxy with Network Security Groups

如果你引入自己的 VNet 并通过网络安全组控制网络流量,则除端口 443 外,还应允许端口 9400 上的入站流量。If you bring your own VNet and control network traffic with network security groups, allow inbound traffic on port 9400 in addition to port 443. 这将确保 Kafka REST 代理服务器可以访问。This will ensure that Kafka REST proxy server is reachable.

先决条件Prerequisites

  1. 将一个应用程序注册到 Azure AD。Register an application with Azure AD. 编写的用来与 Kafka REST 代理交互的客户端应用程序将使用此应用程序的 ID 和机密对 Azure 进行身份验证。The client applications that you write to interact with the Kafka REST proxy will use this application's ID and secret to authenticate to Azure.

  2. 创建 Azure AD 安全组。Create an Azure AD security group. 将已通过 Azure AD 注册的应用程序作为安全组的成员添加到该组中****。Add the application that you've registered with Azure AD to the security group as a member of the group. 此安全组将用于控制允许哪些应用程序与 REST 代理交互。This security group will be used to control which applications are allowed to interact with the REST proxy. 有关创建 Azure AD 组的详细信息,请参阅使用 Azure Active Directory 创建基本组并添加成员For more information on creating Azure AD groups, see Create a basic group and add members using Azure Active Directory.

    验证该组的类型是否为“安全”****。Validate the group is of type Security. 安全组Security Group

    验证该应用程序是否为该组的成员。Validate that application is member of Group. 检查成员身份Check Membership

创建已启用 REST 代理的 Kafka 群集Create a Kafka cluster with REST proxy enabled

以下步骤使用 Azure 门户。The steps below use the Azure portal. 有关使用 Azure CLI 的示例,请参阅使用 Azure CLI 创建 Apache Kafka REST 代理群集For an example using Azure CLI, see Create Apache Kafka REST proxy cluster using Azure CLI.

  1. 在 Kafka 群集创建工作流期间,在“安全 + 网络”选项卡中,选中“启用 Kafka REST 代理”选项 。During the Kafka cluster creation workflow, in the Security + networking tab, check the Enable Kafka REST proxy option.

    启用 Kafka REST 代理并选择安全组

  2. 单击“选择安全组”。****Click Select Security Group. 从安全组列表中,选择你要允许其访问 REST 代理的安全组。From the list of security groups, select the security group that you want to have access to the REST proxy. 可以使用搜索框查找适当的安全组。You can use the search box to find the appropriate security group. 单击底部的“选择”按钮****。Click the Select button at the bottom.

    启用 Kafka REST 代理并选择安全组

  3. 根据使用 Azure 门户在 Azure HDInsight 中创建 Apache Kafka 群集中所述,完成创建群集的剩余步骤。Complete the remaining steps to create your cluster as described in Create Apache Kafka cluster in Azure HDInsight using Azure portal.

  4. 创建群集后,转到群集属性并记下 Kafka REST 代理 URL。Once the cluster is created, go to the cluster properties to record the Kafka REST proxy URL.

    查看 REST 代理 URL

客户端应用程序示例Client application sample

可使用以下 Python 代码来与 Kafka 群集上的 REST 代理交互。You can use the python code below to interact with the REST proxy on your Kafka cluster. 若要使用代码示例,请执行以下步骤:To use the code sample, follow these steps:

  1. 在装有 Python 的计算机上保存示例代码。Save the sample code on a machine with Python installed.

  2. 通过执行 pip3 install msal 安装所需的 Python 依赖项。Install required python dependencies by executing pip3 install msal.

  3. 修改 Configure these properties 代码部分,并更新你的环境的以下属性:Modify the code section Configure these properties and update the following properties for your environment:

    属性Property 说明Description
    租户 IDTenant ID 订阅所在的 Azure 租户。The Azure tenant where your subscription is.
    客户端 IDClient ID 在安全组中注册的应用程序的 ID。The ID for the application that you registered in the security group.
    客户端机密Client Secret 在安全组中注册的应用程序的机密。The secret for the application that you registered in the security group.
    Kafkarest_endpointKafkarest_endpoint 从群集概述的“属性”选项卡中获取此值,如部署部分所述****。Get this value from the Properties tab in the cluster overview as described in the deployment section. 此属性应采用以下格式 – https://<clustername>-kafkarest.azurehdinsight.netIt should be in the following format – https://<clustername>-kafkarest.azurehdinsight.net
  4. 在命令行中,通过执行 sudo python3 <filename.py> 来执行 Python 文件From the command line, execute the python file by executing sudo python3 <filename.py>

此代码执行以下操作:This code does the following action:

  1. 从 Azure AD 提取 OAuth 令牌。Fetches an OAuth token from Azure AD.
  2. 演示如何向 Kafka REST 代理发出请求。Shows how to make a request to Kafka REST proxy.

有关在 Python 中获取 OAuth 令牌的详细信息,请参阅 Python AuthenticationContext 类For more information on getting OAuth tokens in python, see Python AuthenticationContext class. 如果不是通过 Kafka REST 代理创建或删除的 topics 在该处有所反映,则可能会出现延迟。You might see a delay while topics that aren't created or deleted through the Kafka REST proxy are reflected there. 此延迟是因为缓存刷新。This delay is because of cache refresh.

#Required python packages
#pip3 install msal

import msal
import requests

#--------------------------Configure these properties-------------------------------#
# Tenant ID for your Azure Subscription
tenant_id = 'ABCDEFGH-1234-1234-1234-ABCDEFGHIJKL'
# Your Client Application Id
client_id = 'XYZABCDE-1234-1234-1234-ABCDEFGHIJKL'
# Your Client Credentials
client_secret = 'password'
# kafka rest proxy -endpoint
kafkarest_endpoint = "https://<clustername>-kafkarest.azurehdinsight.net"
#--------------------------Configure these properties-------------------------------#

# Scope
scope = 'https://hib.azurehdinsight.net/.default'
#Authority
authority = 'https://login.microsoftonline.com/' + tenant_id

# Create a preferably long-lived app instance which maintains a token cache.
app = msal.ConfidentialClientApplication(
    client_id , client_secret, authority,
    #cache - For details on how look at this example: https://github.com/Azure-Samples/ms-identity-python-webapp/blob/master/app.py
    )

# The pattern to acquire a token looks like this.
result = None

result = app.acquire_token_for_client(scopes=[scope])

print(result)
accessToken = result['access_token']

# relative url
getstatus = "/v1/metadata/topics"
request_url = kafkarest_endpoint + getstatus

# sending get request and saving the response as response object
response = requests.get(request_url, headers={'Authorization': 'Bearer ' + 'accessToken})
print(response.content)

下面是另外一个示例,说明如何使用 curl 命令从 Azure 获取用于 REST 代理的令牌。Find below another sample on how to get a token from Azure for REST proxy using a curl command. 请注意,我们需要在获取令牌时指定 scope=https://hib.azurehdinsight.net/.default****。Notice that we need the scope=https://hib.azurehdinsight.net/.default specified while getting a token.

curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d 'client_id=<clientid>&client_secret=<clientsecret>&grant_type=client_credentials&scope=https://hib.azurehdinsight.net/.default' 'https://login.microsoftonline.com/<tenantid>/oauth2/v2.0/token'

后续步骤Next steps