question

MadugunduSomashekaraRRoopa-1271 avatar image
0 Votes"
MadugunduSomashekaraRRoopa-1271 asked MadugunduSomashekaraRRoopa-1271 commented

After Completion Move option doesn't work

I have a dataflow in pipeline which has .csv as a source file. In the source I have selected After Completion option as Move. My .csv file location is shorttermworkspace/folder1/xyz_.csv. I would like to move the xyz_.csv file from shorttermworkspace/folder1 to shorttermworkspace/folder1/folder2 with same name.

Each time I have to manually create source data .csv file named 'xyz_<timestamp>.csv' inside the folder path 'shorttermworkspace/folder1' where <timestamp> is in yyyyMMddHHmmss format. That's why I used wildcard path instead of giving filename in dataset.

Also, one more information, this sink for this source is not directly connected. I am doing some transformation on this file and use the result of this to do more transformation to finally create a parquet file using sink. Please find the below image for your reference.

189062-moveoption.png

Hope this information will help to reproduce scenario and help me out with a solution.!

Thanks in advance!




azure-data-factoryazure-data-lake-storage
moveoption.png (67.8 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Microsoft Team,

Do I have any update for my above query. Your help is appreciated.

Thanks in advance!

0 Votes 0 ·

Hi @MadugunduSomashekaraRRoopa-1271 ,

Thank you for posting query in Microsoft Q&A Platform.

If I understand ask correctly you want to move files from 'shorttermworkspace/folder1' to 'shorttermworkspace/folder1/folder2'. and i see you already did that configuration. Please correct me if my understanding is wrong.

Any specific error you are facing while files moving?

Kindly elaborate more with simple example on your ask. so that I can help better here. Thank you.

0 Votes 0 ·

Hi @ShaikMaheer-MSFT ,


My .csv file location is shorttermworkspace/folder1/xyz_.csv. I would like to move the xyz_.csv file from shorttermworkspace/folder1 to shorttermworkspace/folder1/folder2 with same name after doing some transformations on this file.

In the source option, though I have given the 'From' and 'To' configuration in After Completion- Move option, my xyz_.csv file is not getting moved from 'shorttermworkspace/folder1' to 'shorttermworkspace/folder1/folder2'.

Could you reproduce the scenario as explained above?

0 Votes 0 ·
MadugunduSomashekaraRRoopa-1271 avatar image MadugunduSomashekaraRRoopa-1271 MadugunduSomashekaraRRoopa-1271 ·

Hi @ShaikMaheer-MSFT,

Did you get a chance to reproduce the scenario as explained above and have any updates for us?

Thanks.

0 Votes 0 ·

0 Answers