question

AnjaliMaithani-7687 avatar image
0 Votes"
AnjaliMaithani-7687 asked ·

How to use Rest API as a Source in Dataflow transformation in ADF

Hello ,

I want to use Web or Rest API as a source in Dataflow transformation and Sink into Azure SQL Database by performing certain transformation like join, lookup and then sink to azure
sql.

Please reply and give suggestion.

Regards,
Anjali Maithani

azure-data-factory
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VaibhavChaudhari avatar image
1 Vote"
VaibhavChaudhari answered ·

You will have to use some intermediate layer for this as I don't see Rest API supported as source in here -

https://docs.microsoft.com/en-us/azure/data-factory/data-flow-source

Probably you can load data using Pipeline Copy activity from API to data lake and then in Data flow Data lake to Azure SQL after transformation


Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered ·

Hello @AnjaliMaithani-7687 ,
Thanks for the ask and using the Microsoft Q&A platform .

Just to add to what @VaibhavChaudhari pointed out , since in your case the sink is SQL , rather then creating a blob you can create a intermediate/staging table in SQL also . Well the advantage with this design is that you will have all the final and intermediate table in SQL , but then you may be loose from CPU power .
Please do let me know how it goes .
Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.