What is the ingestion wizard?
The ingestion wizard makes the data ingestion process easy, fast, and intuitive. Using the ingestion wizard helps you ramp-up quickly to start ingesting data, creating database tables, mapping structures. Select data from different kinds of sources in different data formats, either as a one-time or continuous ingestion process.
The following features make the ingestion wizard so useful:
- Intuitive experience guided by the ingestion wizard
- Ingest data in a matter of minutes
- Ingest data from different kinds of sources: local file, blobs, and containers (up to 10,000 blobs)
- Ingest data in a variety of formats
- Ingest data into new or existing tables
- Table mapping and schema are suggested to you and easy to change
- Continue ingestion easily and quickly from a container with Event Grid
The ingestion wizard is useful when ingesting data for the first time, or when your data's schema is unfamiliar to you.
- An Azure subscription. Create a free Azure account.
- Create a cluster and database.
- Sign in to the Azure Data Explorer web UI and add a connection to your cluster.
To enable access between a cluster and a storage account without public access (restricted to private endpoint/service endpoint), see Create a Managed Private Endpoint.
Access the ingestion wizard
The ingestion wizard guides you through the ingestion process.
To access the wizard from the Azure Data Explorer web UI, use one of the following methods:
To access the ingestion wizard from the Azure Data Explorer home screen in your cluster, complete the first two steps (cluster creation and database creation) and then select Ingest.
To access the wizard from the Azure portal, select Query from the left menu, right-click on the database or table, and select Ingest new data.
This section describes the ingestion wizard in general. The options you select depend on what data format you are ingesting, what kind of data source you are ingesting from, and whether you are ingesting into a new or existing table.
For sample scenarios, see:
The wizard guides you through the following options:
- Ingest into an existing table
- Ingest into a new table
- Ingest data from:
The service automatically generates schema and ingestion properties, which you can change. You can use an existing mapping structure or create a new one, depending on if you're ingesting to a new or existing table.
In the Schema tab, do the following actions:
- Confirm the autogenerated compression type.
- Choose the format of your data. Different formats will allow you to make further changes.
- Change mapping in the Editor window.
The ingestion wizard supports ingesting from source data in all data formats supported by Azure Data Explorer for ingestion.
In the Editor window of the Schema tab, you can adjust data table columns as necessary.
The changes you can make in a table depend on the following parameters:
- Table type is new or existing
- Mapping type is new or existing
|Table type||Mapping type||Available adjustments|
|New table||New mapping||Change data type, Rename column, New column, Delete column, Update column, Sort ascending, Sort descending|
|Existing table||New mapping||New column (on which you can then change data type, rename, and update),
Update column, Sort ascending, Sort descending
|Existing mapping||Sort ascending, Sort descending|
When adding a new column or updating a column, you can change mapping transformations. For more information, see Mapping transformations
At any time, you can open the command editor above the Editor pane. In the command editor, you can view and copy the automatic commands generated from your inputs.
Some data format mappings (Parquet, JSON, and Avro) support simple ingest-time transformations. To apply mapping transformations, create or update a column in the Editor window.
Mapping transformations can be performed on a column of Type string or datetime, with the Source having data type int or long. Supported mapping transformations are:
Once you have completed schema mapping and column manipulations, the ingestion wizard will start the data ingestion process.
When ingesting data from non-container sources, the ingestion will take immediate effect.
If your data source is a container:
Initial data exploration
After ingestion, the wizard gives you options to use Quick commands for initial exploration of your data.
Submit and view feedback for