question

samnick-7321 avatar image
0 Votes"
samnick-7321 asked HimanshuSinha-MSFT commented

DataType in a ForEach loop

Hello,
I currently have a pipeline that takes raw csv files from a blob and loads it into an azure sql DB. Since i have multiple different files, i am using a ForEach loop. The issue here is the datatype. The files being loaded from the blob into the DB are coming over a nvarchar(max) for all fields (even for date, int, float etc). Since its a ForEacch lop and a parameter for file (i.e childitems) , I cannot explicitly mention the mapping for each fields. Is there a way to overcome this or do i have to have a sperate flow for each file and mention the datatype for each fields.

Cheers.

azure-data-factory
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @samnick-7321 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·
HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered

Hello @samnick-7321 ,
Thanks for the ask and using the Microsoft Q&A platform .

I think you are using this pipeline to copy different CSV with varied schema ,, if thats the case , I think your approach is correct . Since you are copying the data to SQL , you can add one stored proc activity and copy the data from the current table to a table with correct data type .


Please do let me know how it goes .
Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

samnick-7321 avatar image
0 Votes"
samnick-7321 answered HimanshuSinha-MSFT commented

@HimanshuSinha-MSFT - But how to incorporate any specific field logic i.e if a specific field in the source file has a value that needs to be updated.
ex: if a field has 123-344-5567 then, the destination field into the DB should be 1233445567

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @samnick-7321, if you want to do data manipulation, you can go with DataFlow. It's not necessary to create different flows for different kind of files, you can have a single DataFlow with multiple SINK in it. Hope this helps. Thanks.

0 Votes 0 ·

Hello @samnick-7321 ,
We haven’t heard from you on the last response from @nasreen-akter and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·