FileAlreadyExistsException: Failed to rename temp file dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/__tmp_path_dir/.2.zip.52d0723f-b803-4a8a-9533-9d6e67813641.tmp to dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/2.zip because file exists

I have built a streaming pipeline with spark autoloader.

Source Folder is a azure blob container.

We encountered a rare issue (could not replicate it). Below is the exception Message:

org.apache.hadoop.fs.FileAlreadyExistsException: Failed to rename temp file dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/__tmp_path_dir/.2.zip.52d0723f-b803-4a8a-9533-9d6e67813641.tmp to dbfs:/mnt/delta_checkpoints/sources/0/rocksdb/2.zip because file exists

Please help on this with a resolution, as this looks like some known platform issue.

PRADEEPCHEEKATLA-MSFT 77,426 Reputation points Microsoft Employee

2022-03-24T09:01:02.987+00:00

Hello @Balasubramanian Singaravelu ,

Welcome to the MS Q&A platform.

From the error message it looks like the temp file already exists. I would request you rename with a new name and see if that works.

In case, if you are experiencing the same error message again. Kindly share the complete command which you are trying to rename temp file.
PRADEEPCHEEKATLA-MSFT 77,426 Reputation points Microsoft Employee

2022-03-28T06:55:33.317+00:00

Hello @Balasubramanian Singaravelu ,

Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.
Hiran Amarathunga 0 Reputation points

2024-04-18T02:53:10.4+00:00

Hello @PRADEEPCHEEKATLA-MSFT

I have the same problem in structured streaming scenario. Will renaming cause some other issues?
Thanks
PRADEEPCHEEKATLA-MSFT 77,426 Reputation points Microsoft Employee

2024-04-19T08:04:21.1366667+00:00

Hiran Amarathunga - Since this thread is too old, I would recommend creating a new thread on the same forum with as much details about your issue as possible. That would make sure that your issue has better visibility in the community.