allow_reuse in Azure ml sdk v2

Rajat Sharma 11

I am running a training pipeline on azure ml python sdk v2. It is running fine but when I am reinitiating it, it takes only 1 second to finish which I assume is taking previous step from last ran pipeline. In v1 sdk we have allow_reuse parameter to stop this behaviour. But how can we do that in ml python sdk v2.

Antoni Czapski 0 Reputation points

2024-02-16T09:09:06.97+00:00

I have a similar issue. The mentioned flag (is_deterministic: false) in the yaml file solves it. But the problem is that my component is in fact deterministic. The only thing that changes is the version. It seems obvious that the pipeline should not reuse the input/output of the component with a changed version, yet it does.

1 answer

PriteeBhuyan-MSFT 16 Reputation points Microsoft Employee

2022-06-29T05:54:50.887+00:00

Hello @Rajat Sharma ,

Thank you for posting query in Microsoft Q&A Platform.

We don't have allow_reuse flag in the v2. However, I would like to understand use case little more in details.
Do you mind sharing why you want to rerun the step when everything is not changed (meaning the output would be exactly the same even if you rerun)?
There is a way to disable reuse behavior by setting "is_deterministic: False" for your component.

Regards,
Pritee
Please sign in to rate this answer.

1 person found this answer helpful.
Rajat Sharma 11 Reputation points

2022-06-29T09:02:48.317+00:00

Hi,

In my scenario I have all my configuration same but my dataset changed and which I am downloading inside the code (every time kind of hardcoded) not passing it as a configurable parameter. So even my configs are same but my dataset has changed, so i want my training to be reinitiated. Below is the way of I am using to create a step for pipeline. Where do I need to set is_deterministic in this function.

data_prep_component = command( name="data_prep_credit_defaults", display_name="Data preparation for training", description="Preprocessing the training data", inputs={ #"input_dir": Input(type="string"), "output_dir": Input(type="string"), }, outputs={ #"output_dir": Output(type="string"), }, # The source folder of the component code=PATH, command="""python datapreprocessor.py \ --output_dir ${{inputs.output_dir}} \ """, environment="bp_environment:3.0", )

PriteeBhuyan-MSFT 16 Reputation points Microsoft Employee

2022-06-30T00:38:17.487+00:00

Hello @Rajat Sharma ,

https://github.com/Azure/azureml-examples/blob/main/sdk/assets/component/train.yml
You can add a is_deterministic property to the yaml definition to make the component non-deterministic thus reuse won't take effect.

Regards,
Pritee

PriteeBhuyan-MSFT 16 Reputation points Microsoft Employee

2022-07-05T22:01:31.157+00:00

Hello @Rajat Sharma ,

Does the comment above for "is_deterministic" property help? Please let us know if you need more help.

Andrew Lemzikov 1 Reputation point

2022-09-20T06:05:00.22+00:00

Hi, I have a similar case where I need a component to run regardless of its inputs each time I run the job. I am using YAML definitions and CLI v2 to deploy my pipeline. is_deterministic: false didn't change anything. Is there any other way to bring back the good old allow_reuse: false functionality?

Andrew Lemzikov 1 Reputation point

2022-09-21T04:13:35.117+00:00

Okay, actually I was wrong. So, specifying is_deterministic: false in a component definition file works well and runs the component regardless of its inputs
component.yml:

name: name_here display_name: display_name_here type: command is_deterministic: false inputs: - ...
Sign in to comment