Use one-click ingestion to create an Event Hub data connection for Azure Data Explorer

• 11/10/2020
• 2 minutes to read
• • o
  • M
  • Y

Azure Data Explorer offers ingestion (data loading) from Event Hubs, a big data streaming platform and event ingestion service. Event Hubs can process millions of events per second in near real-time. In this article, you connect an Event Hub to a table in Azure Data Explorer using the one-click ingestion experience.

Prerequisites

• An Azure account with an active subscription. Create an account for free.
• A cluster and database.
• Event Hub with data for ingestion.

Ingest new data

1. In the left menu of the Web UI, right-click a database or table and select Ingest new data.

The Ingest new data window opens with the Source tab selected.

1. The Database field is auto-populated with your database. You may select a different database from the drop-down menu.

2. Under Table, select Create new and enter a name for the new table, or use an existing table.

   Note

   Table names must be between 1 and 1024 characters. You can use alphanumeric, hyphens, and underscores. Special characters aren't supported.

3. Under Source type, select Reference to Event Hub. The data connect selection will appear.

Data connection

1. Under Data Connection, fill in the following fields:

   

   Setting	Suggested value	Field description
   Data connection name	TestDataConnection	The name that identifies your data connection.
   Subscription		The subscription ID where the Event Hub resource is located.
   Event Hub namespace		The name that identifies your namespace.
   Event Hub		The Event Hub you wish to use.
   Consumer group		The consumer group defined in your Event Hub.
   Data format		Data is read from the Event Hub in form of EventData objects. Supported formats are CSV, JSON, PSV, SCsv, SOHsv TSV, and TSVE.
   Event system properties	Select relevant properties	The Event Hub system properties. If there are multiple records per event message, the system properties will be added to the first one. When adding system properties, create or update table schema and mapping to include the selected properties.

2. Select Edit schema.

Schema tab

For information on schema mapping with JSON-formatted data, see Edit the schema. For information on schema mapping with CSV-formatted data, see Edit the schema.

1. If the data you see in the preview window is not complete, you may need more data to create a table with all necessary data fields. Use the following commands to fetch new data from your Event Hub:

   • Discard and fetch new data: discards the data presented and searches for new events.
   • Fetch more data: Searches for more events in addition to the events already found.

   Note

   To see a preview of your data, your Event Hub must be sending events.

2. Select Start ingestion.

Complete data ingestion

In the Data ingestion completed window, all steps will be marked with green check marks when data ingestion finishes successfully. The cards below these steps give you options to explore your data with Quick queries, undo changes made using Tools, or Monitor the Event Hub connections and data.

Next steps

• Query data in Azure Data Explorer Web UI
• Write queries for Azure Data Explorer using Kusto Query Language