DatasetConsumptionConfig class

Definition

Represent how to deliver the dataset to a compute target.

DatasetConsumptionConfig(name, dataset, mode='direct', path_on_compute=None)
Inheritance
builtins.object
DatasetConsumptionConfig

Parameters

name
str

The name of the dataset in the run, which can be different to the registered name. The name will be registered as environment variable and can be used in data plane.

dataset
Dataset

The dataset that will be consumed in the run.

mode
str

Defines how the dataset should be delivered to the compute target. There are three modes:

  1. 'direct': consume the dataset as dataset.
  2. 'download': download the dataset and consume the dataset as downloaded path.
  3. 'mount': mount the dataset and consume the dataset as mount path.
path_on_compute
str

The target path on the compute to make the data available at. The folder structure of the source data will be kept, however, we might add prefixes to this folder structure to avoid collision. Use tabular_dataset.to_path to see the output folder structure.

Methods

as_download(path_on_compute=None)

Set the mode to download.

as_mount(path_on_compute=None)

Set the mode to mount.

as_download(path_on_compute=None)

Set the mode to download.

as_download(path_on_compute=None)

Parameters

path_on_compute
str

The target path on the compute to make the data available at.

default value: None

Remarks

If path_on_compute starts with a /, then it will be treated as an absolute path. If it doesn't start with a /, then it will be treated as a relative path relative to the working directory. If you have specified an absolute path, please make sure that the job has permission to write to that directory.

as_mount(path_on_compute=None)

Set the mode to mount.

as_mount(path_on_compute=None)

Parameters

path_on_compute
str

The target path on the compute to make the data available at.

default value: None

Remarks

If path_on_compute starts with a /, then it will be treated as an absolute path. If it doesn't start with a /, then it will be treated as a relative path relative to the working directory. If you have specified an absolute path, please make sure that the job has permission to write to that directory.