question

42315514 avatar image
0 Votes"
42315514 asked PRADEEPCHEEKATLA-MSFT commented

Data ingestion is not working in Azure Data Explorer after I created a new data connection

128228-bildschirmfoto-2021-09-01-um-093118.png
I have created a new data connection in my Azure Data Explorer Cluster. I have chosen the connection type Blob storage and the event type Blob created. In addition, I have chosen Resources creation Automatic. I didn't get an error message when I created the data connection. But I get no results when I query my TestDatabase. I used this code to query:
TestTable
| count

The problem is that this code returns "Count": 0, but I have several files in my Blob storage container. I don't know why querying returns "Count": 0. Something is not working with my new data connection.

Why is my data connection not working?

My Blob storage container:
128234-bildschirmfoto-2021-09-01-um-093339.png


azure-blob-storageazure-data-explorerazure-event-grid
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @KimStrasser-4518,

Welcome to the Microsoft Q&A platform.

After creating the data connection have you generating any sample date by creating a blob container because the event will be generated on Blob Created?

Meanwhile, you can checkout this Ingest blobs into Azure Data Explorer by subscribing to Event Grid notifications.


0 Votes 0 ·
42315514 avatar image 42315514 PRADEEPCHEEKATLA-MSFT ·

Yes, I have generated the sample data folder named "31" after the data connection was created:
128525-bildschirmfoto-2021-09-02-um-104222.png
128539-bildschirmfoto-2021-09-02-um-104428.png
The files in my folders end with ".json.gz". I think that I have chosen the wrong settings when I created my data connection.
I have deleted my current data connection. I want to create a new data connection, but I don't know what Basics and Ingest properties that I should use if I want to query all the files in Data Explorer that are located in: mycontainer / PlayFabTitleAllEvents

Should I choose Blob created or Blob renamed in Basics for event type?
Are my filter settings correct?
128642-bildschirmfoto-2021-09-02-um-105843.png

Are my Ingest properties correct? I don't know if I should leave the field Data format empty.
128632-bildschirmfoto-2021-09-02-um-105627.png


0 Votes 0 ·

Hello @KimStrasser-4518,

Thanks for brining this to our attention. As per the repro, we are also observed the same behaviour.

We are reaching out to the internal team to get more details on this behaviour. I will be update you once I hear back from the team.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @KimStrasser-4518,

Thanks for the question and using MS Q&A platform.

I'm able to successfully follow the tutorial: Ingest blobs into Azure Data Explorer by subscribing to Event Grid notifications.

In order to succefully ingest the blobs into ADX you need to modify the Data format to MULTILINE JSON as shown below:

128999-image.png

Now succefully ingest the blobs into ADX.

129096-image.png

In case, if you are still not receive the data with code returns "Count": 0, try running .show ingestion failures to get the details about the failures.

Note: This command returns a result set that includes any ingestion failures that occur when the data ingestion control commands run.

129069-image.png

Hope this helps. Do let us know if you any further queries.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.


image.png (115.9 KiB)
image.png (111.2 KiB)
image.png (89.3 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

It's not working. It still returns "Count": 0. In addition, .show ingestion failures returns no failures:
129267-bildschirmfoto-2021-09-04-um-111157.png

I don't know what I'm doing wrong.
Is it necessary to enter a suffix in Data connection Basics-->Filter settings? I have left this field empty.

My data connection settings and my storage container Blob files:
129316-bildschirmfoto-2021-09-04-um-105944.png
129216-bildschirmfoto-2021-09-04-um-105952.png
129217-bildschirmfoto-2021-09-04-um-110934.png
129298-bildschirmfoto-2021-09-04-um-110903.png


0 Votes 0 ·

Hello @KimStrasser-4518,

When are you uploading the data in to the blob storage account?

After configuring the data connection wait for 5 minutes and then upload the data to blob storage account?

After 10 minutes you can review the data in the ADX. In case you didn't get the data you should get the ingestion failures as shown above?

I would suggest you to follow the steps mentioned in the document: https://docs.microsoft.com/en-us/azure/data-explorer/ingest-data-event-grid#create-an-event-grid-data-connection-in-azure-data-explorer

0 Votes 0 ·
42315514 avatar image 42315514 PRADEEPCHEEKATLA-MSFT ·

Thanx. It works now. I think that I have not waited long enough until I uploaded the files to my storage account.

0 Votes 0 ·
Show more comments