question

MadhuBalan-1479 avatar image
0 Votes"
MadhuBalan-1479 asked MadhuBalan-1479 commented

Azure Data Factory - Load web activity output into a SQL table

I have some data in my SQL table. I have to pass the data in certain columns to a webservice and capture the output of the webservice in some other columns of the same table (Basically the webservice is designed to cleanse my data). I tried using lookup activity and pass the output of the columns in the body of the http request as dynamic content, but it does fetch only 5k rows. Did anybody face such scenarios before ?

azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered MadhuBalan-1479 commented

Hello @MadhuBalan-1479 ,
Thanks for the ask and using the Microsoft Q&A platform .
At this time 5K is the limit for the Lookup(LU) activity and so you are hitting the limit . SInce your source is SQL you use the watermark logic if possible to get around this limit . The watermark is implemented here in this pipeline .


Please do let me know how it goes .
Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @MadhuBalan-1479 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

Thanks @HimanshuSinha-MSFT. That really helps !

0 Votes 0 ·
MohtashamSayeedMohiuddin-8119 avatar image
0 Votes"
MohtashamSayeedMohiuddin-8119 answered MadhuBalan-1479 commented

You have to create a copy activity where you create HTTP Linked Service for the source.

The Sink can be a staging table in the SQL database, and you can create a store procedure for updating existing table.

It is also possible to use Stored Procedure for Sink table.

Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank @MohtashamSayeedMohiuddin-8119 for the input .
@MadhuBalan-1479 L Please do let us know if the input from me or @MohtashamSayeedMohiuddin-8119 helped .
Thanks
Himanshu

1 Vote 1 ·

Thanks @MohtashamSayeedMohiuddin-8119 . This setup is inside my ForEach activity's loop. I have to pass some input to the HTTP body. So, i have to fetch the input from the database using my lookup activity.

1 Vote 1 ·