Hi,
I have a pipeline that copies data from different data sources to ADLS. We run the pipeline (copy activity) and it executes successfully and in copy activity output its shows "file written 1" but when we check that file on ADLS, there is no file.
Do you know how to solve it?
Thanks in advance!
Hello @Pravin007-6871 and welcome to Microsoft Q&A.
As I understand, the copy activity reports 1 written file, but you do not find the expected files?
There are a couple cases I can imagine something kinda similar could happen, but in all cases, something was written, even if not what or where expected.
This would be when parquet or partitioned files or subfolders are involved.
I recall one case where a file named _success was written, to indicate completion. This was a byproduct, not the main data. I suppose if there was no data, the success file would still be written.
Another possibility is it wrote to a subfolder, and the written file is in there. I think subfolder creation is one of the options when partitioning happens.
One more thing, some bits of ADLS can be hidden by certain viewers. Specifically the $logs container when logging is enabled on the storage account. It may be worth checking by another method.
Can you tell me how much data was written, not just the file count?
The copy data activity has written a 4.5Kb file.
In addition, if I rerun the same copy activity(copying the parquet files in the subfolders) at that time, the file has been created successfully in ADLS. It seems that this issue is intermittent.
Is there anything we can do about it?
There is most likely an edge case to explain the behavior, but if not, you can get a deeper investigation either by support ticket, or chat. If you go that route @Pravin007-6871 , it is best to have the runIds of the pipeline runs in which the file is missing. Or their activityRunIds. Having these ID's let the support engineer find the appropriate logs quickly.
For self-investigation, I suggest looking at logs related to the storage account. If a file was indeed written, there should be a record of this.
Or perhaps do a search for any file with update timestamp near the completion of the Copy activity.
Data Factory overwrites files, unlike when the browser downloads and you get myDowload (2).txt . Could overwriting explain your cases?
Does your dataset have a parameterized name or filepath?
Yes we have created the parametrized dataset. Please refer below screenshot.
Also I have checked the other folders also but not found the file.
21 people are following this question.
