question

LaxmiKakarlapudi-8523 avatar image
0 Votes"
LaxmiKakarlapudi-8523 asked ShaikMaheer-MSFT commented

how to pass data from delta lake to rest api in azure

Hi,

I want to build a pipeline where I take data from delta lake and post it to rest api and collect responses.
How do I pass delta lake data dynamically to rest api. Here is the activity I have done :

  1. copy activity where rest is source and sink is adls gen2
    It worked well. But I hard coded records in the body of the source. Now my question is how do I pass it dynamically from delta lake tables.

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

OmarCSiado avatar image
0 Votes"
OmarCSiado answered ShaikMaheer-MSFT commented

Hi @LaxmiKakarlapudi-8523,

I think maybe thinking on batch load schema could work for you, does your API also requires date to load only date related information?, if so, you could create a pipeline parameter and send to the API for example, then also using date for creating folder by date adding files there (here you can find different structures i think depends on your scenario).

Please let us know if that works for you or help us clarifying your scenario with more details so that we will be able to provide suggestions.

Regards,

OCS

· 7
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Omar,

Thank you for your response.
Here is the design that I want to create.
1. copy data from delta lake to adls gen2
2. send the o/p from 1 as input to rest api(post) and collect responses.
3. Save the responses to adls gen2

I am stuck with couple of things here.
For step 1, when I create copy activity to copy data from delta lake to adls gen2 it fails with fs azure account key configuration issue. I did save access key and container in the cluster - advanced settings. But still it is not working.

For step 2, I need to take o/p from step 1 as a body in the rest api activity(post). How can I do that.

For step 3, how to save o/p from step 2 into storage account.



0 Votes 0 ·

Also I need to copy data from delta lake as a json into storage account so that I can pass json data as an input to rest api post activity. How to do that?

0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT LaxmiKakarlapudi-8523 ·

Hi @LaxmiKakarlapudi-8523 ,

You can consider copy activity with Sink data type as json to store data in json format. After that use lookup activity to read data from that json and then pass output of lookup activity to rest api using web activity.

0 Votes 0 ·

Hi,

Thank you for all your responses.

I did try your suggestions and it worked. However it took longer time to process(14 hrs).
I think I need to use streaming. Do anyone have an idea or a video to follow.

Thank you.

0 Votes 0 ·
ShaikMaheer-MSFT avatar image ShaikMaheer-MSFT LaxmiKakarlapudi-8523 ·

Hi @LaxmiKakarlapudi-8523 ,

Thank you for confirming that suggestions worked for you. Could you please mark it as Accepted Answer? Accepted answers helps community as well. Thank you.

0 Votes 0 ·
OmarCSiado avatar image OmarCSiado LaxmiKakarlapudi-8523 ·

Hi @LaxmiKakarlapudi-8523 ,

Thanks for your reply, i think it really depends on the scenario you're facing, there're some variables that have not being considered within your specific question, if you want to do Web API Call then you could use ADF Web Activity component and send parameters for example by Body in JSON format, what i would not recommend is sending a huge amount of data this way, here's a link for documentation:
https://docs.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

On the other hand if what you need is to apply some business rules based on two different data sources i would recommend is following the strategies bellow

  • Create staging/bronce area into datalake then create data flow and add the different data sources to apply business rules there.

  • Create staging area within database then apply business rules via stored proc to get final data into the DB, here you'll need to do stored procedure calls for example to execute your business rules.

Please let us know if that works for you, if so, could you please mark as Accepted Answer?, Accepted answers helps community as well as @ShaikMaheer-MSFT already mentioned.

Thanks!





0 Votes 0 ·

Hi @LaxmiKakarlapudi-8523,

Thank you for marking as accepted answer.

0 Votes 0 ·