MpiStep Class
Creates an Azure ML pipeline step to run an MPI job.
For an example of using MpiStep, see the notebook https://aka.ms/pl-style-trans.
Create an Azure ML pipeline step to run an MPI job.
DEPRECATED. Use the CommandStep instead. For an example see How to run distributed training in pipelines with CommandStep.
- Inheritance
-
MpiStep
Constructor
MpiStep(name=None, source_directory=None, script_name=None, arguments=None, compute_target=None, node_count=None, process_count_per_node=None, inputs=None, outputs=None, allow_reuse=True, version=None, hash_paths=None, **kwargs)Parameters
- source_directory
- str
[Required] A folder that contains Python script, conda env, and other resources used in the step.
- script_name
- str
[Required] The name of a Python script relative to source_directory.
- node_count
- int
[Required] The number of nodes in the compute target used for training. If greater than 1, an mpi distributed job will be run. Only AmlCompute compute target is supported for distributed jobs. PipelineParameter values are supported.
- process_count_per_node
- int
[Required] The number of processes per node. If greater than 1, an mpi distributed job will be run. Only AmlCompute compute target is supported for distributed jobs. PipelineParameter values are supported.
- inputs
- list[Union[InputPortBinding, DataReference, PortDataReference, PipelineData, PipelineOutputAbstractDataset, DatasetConsumptionConfig]]
A list of input port bindings.
A list of output port bindings.
- params
- dict
A dictionary of name-value pairs registered as environment variables with "AML_PARAMETER_".
- allow_reuse
- bool
Indicates whether the step should reuse previous results when re-run with the same settings. Reuse is enabled by default. If the step contents (scripts/dependencies) as well as inputs and parameters remain unchanged, the output from the previous run of this step is reused. When reusing the step, instead of submitting the job to compute, the results from the previous run are immediately made available to any subsequent steps. If you use Azure Machine Learning datasets as inputs, reuse is determined by whether the dataset's definition has changed, not by whether the underlying data has changed.
- version
- str
An optional version tag to denote a change in functionality for the module.
- hash_paths
- list
DEPRECATED: no longer needed.
A list of paths to hash when checking for changes to the step contents. If there
are no changes detected, the pipeline will reuse the step contents from a previous run. By default,
the contents of source_directory is hashed except for files listed in .amlignore or .gitignore.
- use_gpu
- bool
Indicates whether the environment to run the experiment should support GPUs.
If True, a GPU-based default Docker image will be used in the environment. If False, a CPU-based
image will be used. Default docker images (CPU or GPU) will be used only if the custom_docker_image
parameter is not set. This setting is used only in Docker-enabled compute targets.
- use_docker
- bool
Indicates whether the environment to run the experiment should be Docker-based.
- custom_docker_image
- str
The name of the Docker image from which the image to use for training will be built. If not set, a default CPU-based image will be used as the base image.
- user_managed
- bool
Indicates whether Azure ML reuses an existing Python environment; False means that Azure ML will create a Python environment based on the conda dependencies specification.
- conda_packages
- list
A list of strings representing conda packages to be added to the Python environment.
- pip_packages
- list
A list of strings representing pip packages to be added to the Python environment.
- pip_requirements_file_path
- str
The relative path to the pip requirements text file.
This parameter can be specified in combination with the pip_packages parameter.
- environment_definition
- EnvironmentDefinition
The EnvironmentDefinition for the experiment. It includes PythonSection and DockerSection and environment variables. Any environment option not directly exposed through other parameters to the MpiStep construction can be set using environment_definition parameter. If this parameter is specified, it will take precedence over other environment related parameters like use_gpu, custom_docker_image, conda_packages or pip_packages and errors will be reported on these invalid combinations.
- source_directory
- str
[Required] A folder that contains Python script, conda env, and other resources used in the step.
- compute_target
- <xref:azureml.core.compute.AmlComputeCompute>, str
[Required] A compute target to use.
- node_count
- int
[Required] Number of nodes in the compute target used for training. If greater than 1, mpi distributed job will be run. Only AmlCompute compute target is supported for distributed jobs. PipelineParameter values are supported.
- process_count_per_node
- int
[Required] Number of processes per node. If greater than 1, mpi distributed job will be run. Only AmlCompute compute target is supported for distributed jobs. PipelineParameter values are supported.
- inputs
- list[Union[InputPortBinding, DataReference, PortDataReference, PipelineData, PipelineOutputAbstractDataset, DatasetConsumptionConfig]]
A list of input port bindings.
- outputs
- list[Union[PipelineData, OutputDatasetConfig, PipelineOutputAbstractDataset, OutputPortBinding]]
A list of output port bindings.
- params
- dict
A dictionary of name-value pairs registered as environment variables with ">>AML_PARAMETER_<<".
- allow_reuse
- bool
Indicates Whether the step should reuse previous results when re-run with the same parameters remain unchanged, the output from the previous run of this step is reused. When reusing the step, instead of submitting the job to compute, the results from the previous run are immediately made available to any subsequent steps. If you use Azure Machine Learning datasets as inputs, reuse is determined by whether the dataset's definition has changed, not by whether the underlying data has changed.
- hash_paths
- list
DEPRECATED: no longer needed.
A list of paths to hash when checking for changes to the step contents. If there
are no changes detected, the pipeline will reuse the step contents from a previous run. By default,
the contents of source_directory is hashed except for files listed in .amlignore or .gitignore.
- use_gpu
- bool
Indicates whether the environment to run the experiment should support GPUs.
If True, a GPU-based default Docker image will be used in the environment. If False, a CPU-based
image will be used. Default docker images (CPU or GPU) will be used only if the custom_docker_image
parameter is not set. This setting is used only in Docker-enabled compute targets.
- use_docker
- bool
Indicates whether the environment to run the experiment should be Docker-based. custom_docker_image (str): The name of the docker image from which the image to use for mpi job will be built. If not set, a default CPU based image will be used as the base image.
- custom_docker_image
- str
The name of the Docker image from which the image to use for training will be built. If not set, a default CPU-based image will be used as the base image.
- user_managed
- bool
Indicates whether Azure ML reuses an existing Python environment; False means that Azure ML will create a Python environment based on the conda dependencies specification.
- conda_packages
- list
A list of strings representing conda packages to be added to the Python environment.
- pip_packages
- list
A list of strings representing pip packages to be added to the Python environment.
- pip_requirements_file_path
- str
The relative path to the pip requirements text file.
This parameter can be specified in combination with the pip_packages parameter.
- environment_definition
- EnvironmentDefinition
The EnvironmentDefinition for the experiment. It includes PythonSection and DockerSection and environment variables. Any environment option not directly exposed through other parameters to the MpiStep construction can be set using environment_definition parameter. If this parameter is specified, it will take precedence over other environment related parameters like use_gpu, custom_docker_image, conda_packages or pip_packages and errors will be reported on these invalid combinations.
Feedback
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see: https://aka.ms/ContentUserFeedback.
Submit and view feedback for