Indefinite time to run a code which ran in seconds earlier(using data factory)

Question

question

Samyak-3746 asked Apr 25, '22 SumanthMarigowda-MSFT edited May 10, '22

Indefinite time to run a code which ran in seconds earlier(using data factory)

Hi,

I have been following this tutorial:
https://docs.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory

I was able to execute this tutorial successfully earlier but recently, I added ~12 files(all greater than atleast 1Gb), since then I have not been able to run the pipeline successfully and it is taking very long to even run the pipeline.

Once i deleted all the newly loaded files and kept only iris.csv and main.py and ran the pipeline again, it is taking infinitely long to run the code which used to run in seconds earlier.
Also, the node shows in "unusable" state. It shows "Message: The VM disk is full. Delete jobs, tasks, or files on the node to free up space and then reboot the node."

PS: I have a "standard_d2s_v3" machine in my batch account.

Does the job run store all the files of a blob in cache?
What is the solution to this? How to know what is the limit of size/number of files that can be loaded to a storage to run efficiently?

azure-data-factory azure-batch

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · Apr 27 at 11:12 AM

Hello @Samyak-3746,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.

Please don't forget to click on or upvote button whenever the information provided helps you.

0 ·

Answer 1 · 2022-04-26T09:32:16.193Z

PRADEEPCHEEKATLA-MSFT answered Apr 26, '22 PRADEEPCHEEKATLA-MSFT commented May 4, '22

Hello @Samyak-3746,

Thanks for the question and using MS Q&A platform.

As per the node shows “unusable” state means The Compute Node cannot be used for Task execution due to errors.

Cause: There is an increase in memory usage and when that happens the node agent will mark that node as unusable.

Resolution: It is recommended to look at how the task utilizes the disk space. Alternatively, you can choose a larger VM SKU and re-try the same job.

Some of these files are only written once when pool nodes are created, such as pool application packages or pool start task resource files. Even if only written once when the node is created, if these files are too large they could fill the temporary drive.

Other files are written out for each task that is run on a node, such as stdout and stderr. If a large number of tasks run on the same node and/or the task files are too large, they could fill the temporary drive.

To recover an unusable node in VirtualMachineConfiguration pools, you can remove a node from the pool using the remove nodes API. Then, you can grow the pool again to replace the bad node with a fresh one. For CloudServiceConfiguration pools, you can re-image the node via the Batch re-image API. This will clean the entire disk. Re-image is not currently supported for VirtualMachineConfiguration pools.

For more information, refer to Node errors - Node in unusable state & Node disk full.

Hope this will help. Please let us know if any further queries.

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

· 3

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PRADEEPCHEEKATLA-MSFT · Apr 28 at 12:16 PM

Hello @Samyak-3746,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 ·

Samyak-3746 · May 03 at 04:55 PM

Hi @PRADEEPCHEEKATLA-MSFT

Thanks for the response. So, to sum it up, if i have to use the batch again with files in GBs, I might have to use a larger VM? Currently i am using standard_d2s_v3.

0 ·

PRADEEPCHEEKATLA-MSFT Samyak-3746 · May 04 at 05:16 AM

Hello @Samyak-3746,

Yes, you are correct. You can use the larger VM to successfully run the job as per your requirement.

Please don't forget to click on or upvote button whenever the information provided helps you.

0 ·

question