How to use Rest API as a Source in Dataflow transformation in ADF

Anjali Maithani 46 Reputation points
2021-03-05T13:42:01.417+00:00

Hello ,

I want to use Web or Rest API as a source in Dataflow transformation and Sink into Azure SQL Database by performing certain transformation like join, lookup and then sink to azure
sql.

Please reply and give suggestion.

Regards,
Anjali Maithani

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,515 questions
0 comments
Accepted answer
  1. Vaibhav Chaudhari 38,576 Reputation points
    2021-03-05T14:08:40.973+00:00

    You will have to use some intermediate layer for this as I don't see Rest API supported as source in here -

    https://learn.microsoft.com/en-us/azure/data-factory/data-flow-source

    Probably you can load data using Pipeline Copy activity from API to data lake and then in Data flow Data lake to Azure SQL after transformation

    ----------

    Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

    2 people found this answer helpful.

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. HimanshuSinha-msft 19,376 Reputation points Microsoft Employee
    2021-03-11T00:18:44.63+00:00

    Hello @Anjali Maithani ,
    Thanks for the ask and using the Microsoft Q&A platform .

    Just to add to what anonymous userChaudhari pointed out , since in your case the sink is SQL , rather then creating a blob you can create a intermediate/staging table in SQL also . Well the advantage with this design is that you will have all the final and intermediate table in SQL , but then you may be loose from CPU power .
    Please do let me know how it goes .
    Thanks
    Himanshu
    Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

    0 comments