您现在访问的是微软AZURE全睃版技术文档网站,若需覝访问由世纪互蝔违蝥的MICROSOFT AZURE中国区技术文档网站,请访问 https://docs.azure.cn.

将数据从 IoT 中心引入到 Azure 数据资源管理器Ingest data from IoT Hub into Azure Data Explorer

Azure 数据资源管理器是一项快速且高度可缩放的数据探索服务,适用于日志和遥测数据。Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Azure 数据资源管理器提供了从事件中心、IoT 中心和写入 blob 容器的 blob 引入数据(数据加载)的功能。Azure Data Explorer offers ingestion (data loading) from Event Hubs, IoT Hubs, and blobs written to blob containers.

本文介绍如何将数据从 IoT 中心(大数据流处理平台和 IoT 引入服务)引入 Azure 数据资源管理器。This article shows you how to ingest data into Azure Data Explorer from IoT Hub, a big data streaming platform and IoT ingestion service.

有关从 IoT 中心引入到 Azure 数据资源管理器的常规信息,请参阅 连接到 Iot 中心For general information about ingesting into Azure Data Explorer from IoT Hub, see Connect to IoT Hub.

先决条件Prerequisites

创建 IoT 中心Create an Iot Hub

此部分介绍如何使用 Azure 门户创建 IoT 中心。This section describes how to create an IoT hub using the Azure portal.

  1. 登录 Azure 门户Sign in to the Azure portal.

  2. 从 Azure 主页中选择“+ 创建资源”按钮,然后在“搜索市场”字段中输入“IoT 中心”。From the Azure homepage, select the + Create a resource button, and then enter IoT Hub in the Search the Marketplace field.

  3. 在搜索结果中选择“IoT 中心”,然后选择“创建” 。Select IoT Hub from the search results, and then select Create.

  4. 在“基本信息”选项卡上,按如下所示填写字段:On the Basics tab, complete the fields as follows:

    • 订阅:选择要用于中心的订阅。Subscription: Select the subscription to use for your hub.

    • 资源组:选择一个资源组或新建一个资源组。Resource Group: Select a resource group or create a new one. 若要新建资源组,请选择“新建”并填写要使用的名称。To create a new one, select Create new and fill in the name you want to use. 若要使用现有的资源组,请选择它。To use an existing resource group, select that resource group. 有关详细信息,请参阅管理 Azure 资源管理器资源组For more information, see Manage Azure Resource Manager resource groups.

    • 区域:选择中心所在的区域。Region: Select the region in which you want your hub to be located. 选择最靠近你的位置。Select the location closest to you. 某些功能(如 IoT 中心设备流)仅适用于特定区域。Some features, such as IoT Hub device streams, are only available in specific regions. 对于这些受限功能,你必须选择受支持的区域之一。For these limited features, you must select one of the supported regions.

    • IoT 中心名称:输入中心的名称。IoT Hub Name: Enter a name for your hub. 该名称必须全局唯一。This name must be globally unique. 如果输入的名称可用,会显示一个绿色复选标记。If the name you enter is available, a green check mark appears.

    重要

    由于 IoT 中心将作为 DNS 终结点公开可发现,因此请务必避免在命名它时输入任何敏感信息或个人身份信息。Because the IoT hub will be publicly discoverable as a DNS endpoint, be sure to avoid entering any sensitive or personally identifiable information when you name it.

    在 Azure 门户中创建中心

  5. 在完成时选择“下一步:大小和规模”,以继续创建中心。Select Next: Size and scale to continue creating your hub.

    使用 Azure 门户为新的中心设置大小和规模

    可在此处接受默认设置。You can accept the default settings here. 如果需要,可以修改以下任何字段:If desired, you can modify any of the following fields:

    • 定价和缩放层:选择的层。Pricing and scale tier: Your selected tier. 可以根据你需要的功能数以及每天通过解决方案发送的消息数从多个层级中进行选择。You can choose from several tiers, depending on how many features you want and how many messages you send through your solution per day. 免费层适用于测试和评估。The free tier is intended for testing and evaluation. 允许 500 台设备连接到中心,每天最多可传输 8,000 条消息。It allows 500 devices to be connected to the hub and up to 8,000 messages per day. 每个 Azure 订阅可以在免费层中创建一个 IoT 中心。Each Azure subscription can create one IoT hub in the free tier.

      如果正在完成 IoT 中心设备流的快速入门,请选择免费层。If you are working through a Quickstart for IoT Hub device streams, select the free tier.

    • IoT 中心单元:每日每单位允许的消息数取决于中心的定价层。IoT Hub units: The number of messages allowed per unit per day depends on your hub's pricing tier. 例如,如果希望中心支持 700,000 条消息引入,请选择两个 S1 层单位。For example, if you want the hub to support ingress of 700,000 messages, you choose two S1 tier units. 有关其他层选项的详细信息,请参阅选择合适的 IoT 中心层For details about the other tier options, see Choosing the right IoT Hub tier.

    • Azure 安全中心:启用此功能可为 IoT 和设备添加额外的一层威胁防护。Azure Security Center: Turn this on to add an extra layer of threat protection to IoT and your devices. 此选项不可用于免费层的中心。This option is not available for hubs in the free tier. 有关此功能的详细信息,请参阅适用于 IoT 的 Azure 安全中心For more information about this feature, see Azure Security Center for IoT.

    • 高级设置 > 设备到云的分区:此属性将设备到云消息与这些消息的同步读取器数目相关联。Advanced Settings > Device-to-cloud partitions: This property relates the device-to-cloud messages to the number of simultaneous readers of the messages. 大多数中心只需要 4 个分区。Most hubs need only four partitions.

  6. 在完成时选择“下一步:标记”继续到下一屏幕。Select Next: Tags to continue to the next screen.

    标记是名称/值对。Tags are name/value pairs. 可以为多个资源和资源组分配相同的标记,以便对资源进行分类并合并计费。You can assign the same tag to multiple resources and resource groups to categorize resources and consolidate billing. 有关详细信息,请参阅使用标记来组织 Azure 资源for more information, see Use tags to organize your Azure resources.

    使用 Azure 门户为中心分配标记

  7. 在完成时选择“下一步:查看+创建”可查看选择。Select Next: Review + create to review your choices. 你会看到类似于此屏幕的内容,但其中包含创建中心时选择的值。You see something similar to this screen, but with the values you selected when creating the hub.

    查看用于创建新中心的信息

  8. 选择“创建”以创建新的中心。Select Create to create your new hub. 创建中心需要几分钟时间。Creating the hub takes a few minutes.

将设备注册到 IoT 中心Register a device to the IoT Hub

在本部分中,将使用 Azure CLI 为本文创建设备标识。In this section, you use the Azure CLI to create a device identity for this article. 设备 ID 区分大小写。Device IDs are case sensitive.

  1. 打开 Azure Cloud ShellOpen Azure Cloud Shell.

  2. 在 Azure Cloud Shell 中,运行以下命令以安装用于 Azure CLI 的 Microsoft Azure IoT 扩展:In Azure Cloud Shell, run the following command to install the Microsoft Azure IoT Extension for Azure CLI:

    az extension add --name azure-iot
    
  3. 使用以下命令创建一个名为 myDeviceId 的新设备标识并检索设备连接字符串:Create a new device identity called myDeviceId and retrieve the device connection string with these commands:

    az iot hub device-identity create --device-id myDeviceId --hub-name {Your IoT Hub name}
    az iot hub device-identity show-connection-string --device-id myDeviceId --hub-name {Your IoT Hub name} -o table
    

    重要

    收集的日志中可能会显示设备 ID 用于客户支持和故障排除,因此,在为日志命名时,请务必避免包含任何敏感信息。The device ID may be visible in the logs collected for customer support and troubleshooting, so make sure to avoid any sensitive information while naming it.

记下结果中的设备连接字符串。Make a note of the device connection string from the result. 设备应用使用此设备连接字符串以设备身份连接到 IoT 中心。This device connection string is used by the device app to connect to your IoT Hub as a device.

在 Azure 数据资源管理器中创建目标表Create a target table in Azure Data Explorer

现在,在 Azure 数据资源管理器中创建一个表,IoT 中心会向该表发送数据。Now you create a table in Azure Data Explorer to which IoT Hubs will send data. 在群集和数据库(已在先决条件中预配)中创建表。You create the table in the cluster and database provisioned in Prerequisites.

  1. 在 Azure 门户中导航到群集,然后选择“查询”。****In the Azure portal, navigate to your cluster and select Query.

    门户中的 ADX 查询

  2. 将以下命令复制到窗口中,然后选择“运行”**** 以创建将接收引入数据的表 (TestTable)。Copy the following command into the window and select Run to create the table (TestTable) which will receive the ingested data.

    .create table TestTable (temperature: real, humidity: real)
    

    运行创建查询

  3. 将以下命令复制到窗口中,然后选择“运行”**** 将传入的 JSON 数据映射到表 (TestTable) 的列名和数据类型。Copy the following command into the window and select Run to map the incoming JSON data to the column names and data types of the table (TestTable).

    .create table TestTable ingestion json mapping 'TestMapping' '[{"column":"humidity","path":"$.humidity","datatype":"real"},{"column":"temperature","path":"$.temperature","datatype":"real"}]'
    

将 Azure 数据资源管理器表连接到 IoT 中心Connect Azure Data Explorer table to IoT hub

现在,请通过 Azure 数据资源管理器连接到 IoT 中心。Now you connect to the IoT Hub from Azure Data Explorer. 当此连接完成以后,流入 IoT 中心的数据会流式传输到创建的目标表When this connection is complete, data that flows into the iot hub streams to the target table you created.

  1. 在工具栏上选择“通知”****,以验证 IoT 中心部署是否成功。Select Notifications on the toolbar to verify that the IoT Hub deployment was successful.

  2. 在创建的群集下,选择“数据库”****,然后选择已创建的数据库“testdb”****。Under the cluster you created, select Databases then select the database that you created testdb.

    选择测试数据库

  3. 选择“数据引入”****,然后选择“添加数据连接”****。Select Data ingestion and Add data connection.

    创建到 IoT 中心的数据连接-Azure 数据资源管理器

创建数据连接Create a data connection

  1. 使用以下信息填写窗体。Fill out the form with the following information.

    IoT 中心的数据连接窗格-Azure 数据资源管理器

    设置Setting 字段说明Field description
    数据连接名称Data connection name 要在 Azure 数据资源管理器中创建的连接的名称The name of the connection you want to create in Azure Data Explorer
    订阅Subscription 事件中心资源所在的订阅 ID。The subscription ID where the Event Hub resource is located.
    IoT 中心IoT Hub IoT 中心名称IoT Hub name
    共享访问策略Shared access policy 共享访问策略的名称。The name of the shared access policy. 必须有读取权限Must have read permissions
    使用者组Consumer group 在 IoT 中心的内置终结点中定义的使用者组The consumer group defined in the IoT Hub built-in endpoint
    事件系统属性Event system properties IoT 中心事件系统属性The IoT Hub event system properties. 添加系统属性时,创建更新表架构和映射以包括所选属性。When adding system properties, create or update table schema and mapping to include the selected properties.

目标表Target table

路由引入数据有两个选项:静态和动态。****There are two options for routing the ingested data: static and dynamic. 本文将使用静态路由,需在其中指定表名、数据格式和映射。For this article, you use static routing, where you specify the table name, data format, and mapping. 如果事件中心消息包含数据路由信息,此路由信息将覆盖默认设置。If the Event Hub message includes data routing information, this routing information will override the default settings.

  1. 填写以下路由设置:Fill out the following routing settings:

    默认路由属性-IoT 中心-Azure 数据资源管理器

    设置Setting 建议的值Suggested value 字段说明Field description
    表名称Table name TestTableTestTable 在“testdb”**** 中创建的表。The table you created in testdb.
    数据格式Data format JSONJSON 支持的格式为 Avro、CSV、JSON、MULTILINE JSON、ORC、PARQUET、PSV、SCSV、SOHSV、TSV、TXT、TSVE、APACHEAVRO 和 W3CLOG。Supported formats are Avro, CSV, JSON, MULTILINE JSON, ORC, PARQUET, PSV, SCSV, SOHSV, TSV, TXT, TSVE, APACHEAVRO, and W3CLOG.
    映射Mapping TestMapping**TestMapping testdb中创建的映射,用于将传入的数据映射到testdb的列名称和数据类型。The mapping you created in testdb, which maps incoming data to the column names and data types of testdb. 对于 JSON、多行 JSON 和 AVRO 是必需的,对于其他格式是可选的。Required for JSON, MULTILINE JSON, and AVRO, and optional for other formats.

    警告

    如果进行手动故障转移,必须重新创建数据连接。In case of a manual failover, you must recreate the data connection.

    备注

    • 无需指定所有 默认路由设置You don't have to specify all Default routing settings. 还接受部分设置。Partial settings are also accepted.
    • 只有创建数据连接后进入队列的事件才会被引入。Only events enqueued after you create the data connection are ingested.
  2. 选择“创建”。Select Create.

事件系统属性映射Event system properties mapping

备注

  • 单记录事件支持系统属性。System properties are supported for single-record events.
  • 对于 csv 映射,属性将添加到记录的开头。For csv mapping, properties are added at the beginning of the record. 对于 json 映射,将根据下拉列表中显示的名称添加属性。For json mapping, properties are added according to the name that appears in the drop-down list.

如果在表的 "数据源" 部分中选择了 "事件系统属性",则必须在表架构和映射中包含系统属性If you selected Event system properties in the Data Source section of the table, you must include system properties in the table schema and mapping.

生成用于测试的示例数据Generate sample data for testing

模拟设备应用程序会连接到 IoT 中心上特定于设备的终结点,并发送模拟的温度和湿度遥测数据。The simulated device application connects to a device-specific endpoint on your IoT hub and sends simulated temperature and humidity telemetry.

  1. https://github.com/Azure-Samples/azure-iot-samples-csharp/archive/master.zip 下载示例 C# 项目并提取 ZIP 存档。Download the sample C# project from https://github.com/Azure-Samples/azure-iot-samples-csharp/archive/master.zip and extract the ZIP archive.

  2. 在本地终端窗口中,导航到示例 C# 项目的根文件夹。In a local terminal window, navigate to the root folder of the sample C# project. 然后导航到 iot-hub\Quickstarts\simulated-device 文件夹。Then navigate to the iot-hub\Quickstarts\simulated-device folder.

  3. 在所选文本编辑器中打开 SimulatedDevice.cs 文件。Open the SimulatedDevice.cs file in a text editor of your choice.

    s_connectionString 变量的值替换为将设备注册到 IoT 中心中的设备连接字符串。Replace the value of the s_connectionString variable with the device connection string from Register a device to the IoT Hub. 然后将更改保存到 SimulatedDevice.cs 文件****。Then save your changes to SimulatedDevice.cs file.

  4. 在本地终端窗口中,运行以下命令以安装模拟设备应用程序所需的包:In the local terminal window, run the following commands to install the required packages for simulated device application:

    dotnet restore
    
  5. 在本地终端窗口中,运行以下命令,生成并运行模拟设备应用程序:In the local terminal window, run the following command to build and run the simulated device application:

    dotnet run
    

    以下屏幕截图显示了模拟设备应用程序将遥测数据发送到 IoT 中心后的输出:The following screenshot shows the output as the simulated device application sends telemetry to your IoT hub:

    运行模拟设备

查看数据流Review the data flow

应用生成数据以后,现在可以看到该数据从 IoT 中心流到群集中的表。With the app generating data, you can now see the data flow from the IoT hub to the table in your cluster.

  1. 在 Azure 门户中的 IoT 中心下,可以看到应用运行时活动的峰值。In the Azure portal, under your IoT hub, you see the spike in activity while the app is running.

    IoT 中心度量值

  2. 若要检查到目前为止已向数据库发送的消息数,请在测试数据库中运行以下查询。To check how many messages have made it to the database so far, run the following query in your test database.

    TestTable
    | count
    
  3. 若要查看消息的内容,请运行以下查询:To see the content of the messages, run the following query:

    TestTable
    

    结果集:The result set:

    显示引入数据结果

    备注

    • Azure 数据资源管理器具有用于数据引入的聚合(批处理)策略,旨在优化引入过程。Azure Data Explorer has an aggregation (batching) policy for data ingestion, designed to optimize the ingestion process. 默认情况下,该策略配置为 5 分钟或 500 MB 数据,因此你可能会遇到延迟。The policy is configured to 5 minutes or 500 MB of data, by default, so you may experience a latency. 有关聚合选项,请参阅批处理策略See batching policy for aggregation options.
    • 配置表以支持流式处理并消除响应时间延迟。Configure your table to support streaming and remove the lag in response time. 请参阅流式处理策略See streaming policy.

清理资源Clean up resources

如果不打算再次使用 IoT 中心,请清理资源组以避免产生费用。If you don't plan to use your IoT Hub again, clean up your resource group to avoid incurring costs.

  1. 在 Azure 门户的最左侧选择“资源组”,,然后选择创建的资源组。****In the Azure portal, select Resource groups on the far left, and then select the resource group you created.

    如果左侧菜单处于折叠状态,请选择If the left menu is collapsed, select “展开”按钮 将其展开。to expand it.

    选择要删除的资源组

  2. 在“test-resource-group”**** 下,选择“删除资源组”****。Under test-resource-group, select Delete resource group.

  3. 在新窗口中键入要删除的资源组的名称,然后选择“删除”。In the new window, type the name of the resource group to delete it, and then select Delete.

后续步骤Next steps