question

SampatVarun-0080 avatar image
0 Votes"
SampatVarun-0080 asked whoward-msft edited

How to delete the local copy of an uploaded blob on Python, Azure Batch Services.

Hello.

I recently followed the attached tutorial, and it worked perfectly with the provided code. (link: https://docs.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory)

I realised that Pandas stores a local copy on the VMs used as part of the Batch Service, so I thought it might be good practise to delete the file just uploaded, and tried the following piece of code:

 container_client = blob_service_client.get_container_client(containerName)
 with open(output_file_name, "rb") as data:
     blob_client = container_client.upload_blob(name = output_file_name, 
     data=data)
     if os.path.exists(output_file_name):
         os.remove(output_file_name)
     else:
         pass

However, my batch output error file (stderr.txt) returned the following error:

  • Traceback (most recent call last):
    File "main.py", line 44, in <module>
    os.remove(output_file_name)
    PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'iris_setosa.csv'


What should I do? Is there a way to delete files? If not, what is the best alternative? OR am I just confused about how Azure Batch Services work? Does the file get deleted automatically?

If I am wrong, I apologise in advance, since it's my first time working with Batch Services.

Thank you in advance!

azure-data-factoryazure-batch
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

whoward-msft avatar image
0 Votes"
whoward-msft answered whoward-msft edited

Hi @SampatVarun-0080,

Thanks for the well written question. Firstly the error message your getting is because the file is still being used by the other process. Amend your script to remove the file after the other process is done with it. This can be done by putting it outside the the with keyword.

  container_client = blob_service_client.get_container_client(containerName)
  with open(output_file_name, "rb") as data:
      blob_client = container_client.upload_blob(name = output_file_name, 
      data=data)
    
  if os.path.exists(output_file_name):
          os.remove(output_file_name)
  else:
          pass

Now, it would be easier to simply remove the file from the Blob using Azure Storage Explorer since you had to have set that up as part of this exercise.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.