question

HimanshuGautam-0592 avatar image
0 Votes"
HimanshuGautam-0592 asked ramr-msft answered

Pre-existing Compute Resource necessary for running a scheduled ML pipeline?

Hi fellows,
I have been exploring Azure ML Pipeline. I am referring to this notebook for the below code:

Here is a small snippet from a MS Repo:
train_step = PythonScriptStep(name = "Prepare Data",
source_directory = experiment_folder,
script_name = "prep_diabetes.py",
arguments = ['--input-data', diabetes_ds.as_named_input('raw_data'),
'--prepped-data', prepped_data_folder],
outputs=[prepped_data_folder],
compute_target = pipeline_cluster,
runconfig = pipeline_run_config,
allow_reuse = True)

This suggests that while defining a pipeline, we must provide it a compute resource. This obviously makes sense, since specific compute might be required for a specific step.

But do we need to have this compute resource up and running always, so that whenever a pipeline is triggered, it can find the compute resource?

Also, i figured we can probably keep a cluster with Zero minimum nodes, in which cases cluster is resized whenever pipeline is triggered. But i think there is a minimal cost incurrent in probably container registry regularly in such a setup. Is this the recommended way to deploy ML pipelines or some more efficient approach is possible?


azure-machine-learning
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

ramr-msft avatar image
0 Votes"
ramr-msft answered

@HimanshuGautam-0592 Thanks for the question. AmlCompute clusters are designed to scale dynamically based on your workload. The cluster can be scaled up to the maximum number of nodes you configure. As each run completes, the cluster will release nodes and scale to your configured minimum node count.

To avoid charges when no jobs are running, set the minimum nodes to 0. This setting allows Azure Machine Learning to de-allocate the nodes when they aren't in use. Any value larger than 0 will keep that number of nodes running, even if they are not in use.


Another way to save money on compute resources is Azure Reserved VM Instance. amlcompute supports reserved instances out of the box. These reservations can be used across azure compute resources (vmss/batch/vm) and AzureML compute.

Check out this article to learn more about planning and managing costs : https://docs.microsoft.com/en-us/azure/machine-learning/concept-plan-manage-cost


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.