question

Cricket-7789 avatar image
0 Votes"
Cricket-7789 asked ·

Azure Data Factory (ADF) Webhook activity response ingestion into azure storage container.

We are using Azure Data Factory and Databricks for building data lake, we are receiving data from various sources and uses Azure Data Factory (ADF) to ingest data in azure container storage.

Recently, one of the source system is sharing data as a Webhooks and there are around number of Webhooks which we need to listen. The Webhooks sends data in JSON format.

I am trying to configure Webbook activity in the Azure Data Factory, but I am not finding any option to write the data in Container storage.

If anybody has configured Webhooks into ADF and storing the response in azure storage containers, please share your approach or configuration.

Also, please share the best practices and use cases of webhooks.

Thank you.

azure-data-factoryazure-storage-accounts
· 4
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@cricket-7789 Thanks for using Microsoft Q&A !!

I have not implemented WebHook activity but as per my knowledge WebHook activity only supports POST in ADF and you can send payload in the request body. Is it an Azure WebHook ? Can you please elaborate how data is being received via WebHooks in your case ?
If your endpoints are REST based then you can probably use Copy Data activity with REST as a source to get the required data in ADF and then send it to respective Azure storage.
Thanks
Saurabh



0 Votes 0 ·
Cricket-7789 avatar image Cricket-7789 SaurabhSharma-msft ·

Hello @SaurabhSharma-msft ,

Thank you for your response and sorry for the delay.

We are now able to get events from Webhook using Azure Function by writing the code in python.

The job of the function is to read Webhook events and save data into azure storage. We are using an HTTP trigger template to get the job done and able to receive the events from the Webhook, but when I try to write to azure storage it is giving me below error.

System.InvalidOperationException: Storage account connection string 'AzureWebJobs<AzureStorageAccountName>' does not exist. Make sure that it is a defined App Setting.

Can you please help here how to push data into azure storage from Azure Function.

0 Votes 0 ·

@cricket-7789 sorry for the delay but have you defined AzureWebJobs settings ?


0 Votes 0 ·
Show more comments

0 Answers