將資料從事件中樞內嵌至 Azure 資料總管Ingest data from Event Hub into Azure Data Explorer

Azure 資料總管是一項快速又可高度調整的資料探索服務,可用於處理記錄和遙測資料。Azure Data Explorer is a fast and highly scalable data exploration service for log and telemetry data. Azure 資料總管可從事件中樞、巨量資料串流平台及事件內嵌服務進行內嵌 (載入資料)。Azure Data Explorer offers ingestion (data loading) from Event Hubs, a big data streaming platform and event ingestion service. 事件中樞可以近乎即時地每秒鐘處理數百萬個事件。Event Hubs can process millions of events per second in near real-time. 在本文中,您可以建立事件中樞,從 Azure 資料總管和流程通過系統時,請參閱資料連接到它。In this article, you create an event hub, connect to it from Azure Data Explorer and see data flow through the system.

必要條件Prerequisites

登入 Azure 入口網站Sign in to the Azure portal

登入 Azure 入口網站Sign in to the Azure portal.

建立事件中樞Create an event hub

在本文中,您會產生範例資料,並將它傳送到事件中樞。In this article, you generate sample data and send it to an event hub. 第一個步驟是建立事件中樞。The first step is to create an event hub. 其做法是使用 Azure 入口網站中的 Azure Resource Manager 範本。You do this by using an Azure Resource Manager template in the Azure portal.

  1. 若要建立事件中樞,請使用下列按鈕開始部署。To create an event hub, use the following button to start the deployment. 按一下滑鼠右鍵並選取 [在新視窗中開啟] ,以便依照本文中的其餘步驟操作。Right-click and select Open in new window, so you can follow the rest of the steps in this article.

    部署至 AzureDeploy to Azure

    [部署至 Azure] 按鈕可將您帶往 Azure 入口網站,填寫部署表單。The Deploy to Azure button takes you to the Azure portal to fill out a deployment form.

    部署至 Azure

  2. 選取用於建立事件中樞的訂用帳戶,然後建立一個名為 test-hub-rg 的資源群組。Select the subscription where you want to create the event hub, and create a resource group named test-hub-rg.

    建立資源群組

  3. 在表單中填寫以下資訊。Fill out the form with the following information.

    部署表單

    對於下表中未列出的任何設定,請使用預設值。Use defaults for any settings not listed in the following table.

    設定Setting 建議的值Suggested value 欄位描述Field description
    訂用帳戶Subscription 您的訂用帳戶Your subscription 選取您要用於事件中樞的 Azure 訂用帳戶。Select the Azure subscription that you want to use for your event hub.
    資源群組Resource group test-hub-rgtest-hub-rg 建立新的資源群組。Create a new resource group.
    位置Location 美國西部 West US 選取 美國西部這篇文章。Select West US for this article. 至於生產系統,請選取最符合您需求的區域。For a production system, select the region that best meets your needs. 將事件中樞命名空間建立在與 Kusto 相同的 [位置] 可獲得最佳效能 (對於高輸送量的事件中樞命名空間格外重要)。Create the event hub namespace in the same Location as the Kusto cluster for best performance (most important for event hub namespaces with high throughput).
    命名空間名稱Namespace name 唯一命名空間名稱A unique namespace name 選擇可識別您命名空間的唯一名稱。Choose a unique name that identifies your namespace. 例如,mytestnamespaceFor example, mytestnamespace. 網域名稱 servicebus.windows.net 已附加至您提供的名稱。The domain name servicebus.windows.net is appended to the name you provide. 名稱只能包含字母、數字和連字號。The name can contain only letters, numbers, and hyphens. 名稱必須以字母開頭,且必須以字母或數字結尾。The name must start with a letter, and it must end with a letter or number. 此值長度必須介於 6 至 50 個字元之間。The value must be between 6 and 50 characters long.
    事件中樞名稱Event hub name test-hubtest-hub 事件中樞位於命名空間之下,其會提供專屬的唯一範圍容器。The event hub sits under the namespace, which provides a unique scoping container. 事件中樞名稱在命名空間內不可重複。The event hub name must be unique within the namespace.
    取用者群組名稱Consumer group name test-grouptest-group 取用者群組能讓多個取用應用程式各自擁有獨立的事件串流檢視。Consumer groups enable multiple consuming applications to each have a separate view of the event stream.
  4. 選取 [採購] ,這會確認您正在您的訂用帳戶中建立資源。Select Purchase, which acknowledges that you're creating resources in your subscription.

  5. 在工具列上選取 [通知] 以監視佈建程序。Select Notifications on the toolbar to monitor the provisioning process. 可能需要幾分鐘的時間,部署才會成功,但您現在可以移至下一個步驟。It might take several minutes for the deployment to succeed, but you can move on to the next step now.

    通知

在 Azure 資料總管中建立目標資料表Create a target table in Azure Data Explorer

現在要在 Azure 資料總管中建立一個資料表,供事件中樞將資料傳送至此。Now you create a table in Azure Data Explorer, to which Event Hubs will send data. 您會在於必要條件中佈建的叢集與資料庫中建立該資料表。You create the table in the cluster and database provisioned in Prerequisites.

  1. 在 Azure 入口網站中瀏覽至您的叢集,然後選取 [查詢] 。In the Azure portal, navigate to your cluster then select Query.

    查詢應用程式連結

  2. 將下列命令複製到視窗,然後選取 [執行] 以建立資料表 (TestTable),該資料表會接收內嵌的資料。Copy the following command into the window and select Run to create the table (TestTable) which will receive the ingested data.

    .create table TestTable (TimeStamp: datetime, Name: string, Metric: int, Source:string)
    

    執行建立查詢

  3. 將下列命令複製到視窗中,然後選取 [執行] 以將傳入的 JSON 資料對應至資料表 (TestTable) 的資料行名稱與資料類型。Copy the following command into the window and select Run to map the incoming JSON data to the column names and data types of the table (TestTable).

    .create table TestTable ingestion json mapping 'TestMapping' '[{"column":"TimeStamp","path":"$.timeStamp","datatype":"datetime"},{"column":"Name","path":"$.name","datatype":"string"},{"column":"Metric","path":"$.metric","datatype":"int"},{"column":"Source","path":"$.source","datatype":"string"}]'
    

連線至事件中樞Connect to the event hub

現在您應從 Azure 資料總管連線至事件中樞。Now you connect to the event hub from Azure Data Explorer. 此連線建立後,流入事件中樞資料的資料即會串流至您先前在本文中建立的測試資料表。When this connection is in place, data that flows into the event hub streams to the test table you created earlier in this article.

  1. 選取工具列上的 [通知] ,以確認事件中樞部署已成功。Select Notifications on the toolbar to verify that the event hub deployment was successful.

  2. 在您建立的叢集下方,選取 [資料庫] ,然後選取 [TestDatabase] 。Under the cluster you created, select Databases then TestDatabase.

    選取測試資料庫

  3. 選取 [資料擷取] ,然後選取 [新增資料連線] 。Select Data ingestion and Add data connection. 然後,在表單中填寫以下資訊。Then fill out the form with the following information. 完成後,選取 [建立] 。Select Create when you are finished.

    事件中樞連線

    資料來源:Data Source:

    設定Setting 建議的值Suggested value 欄位描述Field description
    資料連線名稱Data connection name test-hub-connectiontest-hub-connection 您想要在 Azure 資料總管中建立的連線名稱。The name of the connection you want to create in Azure Data Explorer.
    事件中樞命名空間Event hub namespace 唯一命名空間名稱A unique namespace name 您先前選擇用來辨識命名空間的名稱。The name you chose earlier that identifies your namespace.
    事件中樞Event hub test-hubtest-hub 您建立的事件中樞。The event hub you created.
    取用者群組Consumer group test-grouptest-group 在您所建立事件中樞中定義的取用者群組。The consumer group defined in the event hub you created.

    目標資料表:Target table:

    路由內嵌資料有兩個選項:靜態 和動態 。There are two options for routing the ingested data: static and dynamic. 在本文中,您將使用靜態路由,您會指定資料表名稱、資料格式和對應。For this article, you use static routing, where you specify the table name, data format, and mapping. 因此,將 [我的資料包含路由資訊] 保留為未選取。Therefore, leave My data includes routing info unselected.

    設定Setting 建議的值Suggested value 欄位描述Field description
    資料表Table TestTableTestTable 您在 TestDatabase 中建立的資料表。The table you created in TestDatabase.
    資料格式Data format JSONJSON 支援的格式為 Avro、CSV、JSON、MULTILINE JSON、PSV、SOH、SCSV、TSV 和 TXT。Supported formats are Avro, CSV, JSON, MULTILINE JSON, PSV, SOH, SCSV, TSV, and TXT.
    資料行對應Column mapping TestMappingTestMapping 您在 TestDatabase 中建立的對應,會將傳入的 JSON 資料對應至 TestTable 的資料行名稱與資料類型。The mapping you created in TestDatabase, which maps incoming JSON data to the column names and data types of TestTable. 需要 JSON、MULTILINE JSON 或 AVRO,而其他格式為選用性質。Required for JSON, MULTILINE JSON, or AVRO, and optional for other formats.

    注意

    選取 [我的資料包含路由資訊] 來使用動態路由,其中您的資料會包含必要的路由資訊,如範例應用程式註解中所示。Select My data includes routing info to use dynamic routing, where your data includes the necessary routing information as seen in the sample app comments. 如果靜態和動態屬性都已設定,則動態屬性會覆寫靜態屬性。If both static and dynamic properties are set, the dynamic properties override the static ones.

複製連接字串Copy the connection string

當您執行「必要條件」中所列的範例應用程式時,您需要事件中樞命名空間的連接字串。When you run the sample app listed in Prerequisites, you need the connection string for the event hub namespace.

  1. 在您建立的事件中樞命名空間下方,選取 [共用存取原則] ,然後選取 [RootManageSharedAccessKey] 。Under the event hub namespace you created, select Shared access policies, then RootManageSharedAccessKey.

    共用存取原則

  2. 複製 [連接字串 - 主索引鍵] 。Copy Connection string - primary key. 您在下一節中貼上這項資料。You paste it in the next section.

    連接字串

產生範例資料Generate sample data

使用下載的範例應用程式來產生資料。Use the sample app you downloaded to generate data.

  1. 請在 Visual Studio 中開啟範例應用程式解決方案。Open the sample app solution in Visual Studio.

  2. 在 [program.cs] 檔案中,將 connectionString 常數更新為您從事件中樞命名空間複製的連接字串。In the program.cs file, update the connectionString constant to the connection string you copied from the event hub namespace.

    const string eventHubName = "test-hub";
    // Copy the connection string ("Connection string-primary key") from your Event Hub namespace.
    const string connectionString = @"<YourConnectionString>";
    
  3. 建置並執行應用程式。Build and run the app. 應用程式會將訊息傳送至事件中樞,會每隔 10 秒列印出狀態。The app sends messages to the event hub, and it prints out status every ten seconds.

  4. 在應用程式傳送一些訊息後,請移至下一步:檢閱流入您事件中樞與測試資料表的資料流程。After the app has sent a few messages, move on to the next step: reviewing the flow of data into your event hub and test table.

檢閱資料流程Review the data flow

在應用程式產生資料後,您現在可以檢視該資料從事件中樞傳至叢集中資料表的流程。With the app generating data, you can now see the flow of that data from the event hub to the table in your cluster.

  1. 當應用程式正在執行時,在 Azure 入口網站內事件中樞的下方,您會看見活動爆增。In the Azure portal, under your event hub, you see the spike in activity while the app is running.

    事件中樞圖

  2. 若要檢查目前為止已有多少則訊息成功進入資料庫,請在測試資料庫中執行下列查詢。To check how many messages have made it to the database so far, run the following query in your test database.

    TestTable
    | count
    
  3. 若要查看訊息的內容,請執行下列查詢:To see the content of the messages, run the following query:

    TestTable
    

    結果集應會如下所示:The result set should look like the following:

    訊息結果集

    注意

    Azure 資料總管具有資料擷取的彙總 (批次處理) 原則,可將擷取程序最佳化。Azure Data Explorer has an aggregation (batching) policy for data ingestion, designed to optimize the ingestion process. 此原則已設定為 5 分鐘,因此您可能會遇到延遲。The policy is configured to 5 minutes so you may experience a latency.

清除資源Clean up resources

如果您不打算再次使用您的事件中樞,請清除 test-hub-rg 以避免產生成本。If you don't plan to use your event hub again, clean up test-hub-rg, to avoid incurring costs.

  1. 在 Azure 入口網站中選取靠左側的 [資源群組] ,然後選取您所建立的群組。In the Azure portal, select Resource groups on the far left, and then select the resource group you created.

    如果左側功能表已摺疊,請選取If the left menu is collapsed, select [展開] 按鈕 加以展開。to expand it.

    選取要刪除的資源群組

  2. 在 [test-resource-group] 下方,選取 [刪除資源群組] 。Under test-resource-group, select Delete resource group.

  3. 在新視窗中,輸入要刪除的資源群組名稱 (test-hub-rg),然後選取 [刪除] 。In the new window, type the name of the resource group to delete (test-hub-rg), and then select Delete.

後續步驟Next steps