question

VigneshwarSB-6839 avatar image
0 Votes"
VigneshwarSB-6839 asked HimanshuSinha-MSFT commented

ETL using Azure Data Factory

Hi,

I need to implement an ETL using Azure Data Factory. Can someone help me out here?

Use Case:
1. Read data from SQL server using Lookup activity.
2. Use a For Each loop to iterate through each row and call a web service that helps with the transformation of data and returns transformed data.
3. Now I need to append each of these rows with the transformed data and write them into a CSV file.

I have completed the first 2 steps. Stuck in 3rd part.

Can someone help me out on how to create a CSV file after the transformation? Or I welcome if anyone has a better approach to this.

Thanks in advance.





azure-data-factoryazure-functions
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered HimanshuSinha-MSFT commented

@VigneshwarSB-6839 : Since you have already achieved #1 & #2 , I think you can write the response from the WS to a new csv file and then use mapping data flow . In the mapping data flow you can join these two stream . ( assming that you atleast have one column which you can use in a join)

Thanks
Himanshu

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@HimanshuSinha-MSFT Web service call is being from inside a ForEach loop. I don't see any way to write the response from my WS into a CSV file.

I need to write each transformed row(which I get back as a response from WS) into CSV file. How can I do that ?

0 Votes 0 ·

Hello @VigneshwarSB-6839 ,

Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.

Thanks
Himanshu

0 Votes 0 ·

Hello @VigneshwarSB-6839 , ,
We haven’t received any email from you as requested in the last response and was just checking back to see if you have a resolution.In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, please respond back with the more details as requested and we will try to help .
Thanks
Himanshu

0 Votes 0 ·
Kiran-MSFT avatar image
0 Votes"
Kiran-MSFT answered HimanshuSinha-MSFT converted comment to answer

ETL is best implemented using dataflow for scale and performance and use pipeline for orchestration.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@Kiran-MSFT I'm new to the Data factory and Data flows. Can you explain me how can I call my web service from inside Data flow for each row of data and then accumulate them back into a CSV file ?

0 Votes 0 ·