question

RoscoeLewis-7527 avatar image
0 Votes"
RoscoeLewis-7527 asked PramodValavala-MSFT commented

Azure intergration of API to data lake using service bus topic

I am building an integration using API's from ADP. There is already an endpoint that I will be using - this is tied to a azure service bus. I have the name of the topic but I'm having trouble understanding the next artifacts that need to be created in Azure. I want the incoming messages to hit an EDW or just the data lake as data (it might be coming in as XML format, which I may need to convert to azure sql database):

Should I create another subscription tied to the EDW that can pick up the messages from the service bus topic? (I have not attempted this before)
Or Should I create a logic app to directly read the service bus or service bus topic? (I have not attempted this either)
I need to design a scalable solution - any insight would be greatly appreciated

Thanks

azure-api-managementazure-service-bus
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PramodValavala-MSFT avatar image
0 Votes"
PramodValavala-MSFT answered PramodValavala-MSFT commented

Azure Functions might be a better approach to consider. You could leverage the Service Bus Binding to trigger your function (which automatically scales out based on volume of messages).

Later in code, you could fetch the data, convert, and store in SQL.


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you ! - can you expand on this answer or is there documentation to see how to linked the service bus topic to azure function then dump into data lake?

0 Votes 0 ·

The Service Bus Trigger Reference should cover how the binding is setup. As for data lake if you are using Azure Data Lake Storage Gen2, you could leverage the Blob Storage Binding as well (refer this answer for more info).

I'm unaware of an end-to-end doc for this scenario but I believe the references should help.


1 Vote 1 ·