DataReference Class

Represents a reference to data in a datastore.

A DataReference represents a path in a datastore and can be used to describe how and where data should be made available in a run. It is no longer the recommended approach for data access and delivery in Azure Machine Learning. Dataset supports accessing data from Azure Blob storage, Azure Files, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Azure SQL Database, and Azure Database for PostgreSQL through unified interface with added data management capabilities. It is recommended to use dataset for reading data in your machine learning projects.

For more information on how to use Azure ML dataset in two common scenarios, see the articles:

Inheritance
DataReference

Constructor

DataReference(datastore, data_reference_name=None, path_on_datastore=None, mode='mount', path_on_compute=None, overwrite=False)

Parameters

datastore
AbstractAzureStorageDatastore or AzureDataLakeDatastore

The datastore to reference.

data_reference_name
str

The name of the data reference.

path_on_datastore
str

The relative path in the backing storage for the data reference.

mode
str

The operation on the data reference. Supported values are 'mount' (the default) and 'download'.

Use the 'download' mode when your script expects a specific (e.g., hard-coded) path for the input data. In this case, specify the path with the path_on_compute parameter when you declare the DataReference. Azure Machine Learning will download the data specified by that path before executing your script.

With the 'mount' mode, a temporary directory is created with the mounted data and an environment variable $AZUREML_DATAREFERENCE_<data_reference_name> is set with the path to the temporary directory. If you pass a DataReference into the arguments list for a pipeline step (e.g. PythonScriptStep), then the reference will be expanded to the local data path at runtime.

path_on_compute
str

The path on the compute target for the data reference.

overwrite
bool

Indicates whether to overwrite existing data.

Remarks

A DataReference defines both the data location and how the data is used on the target compute binding (mount or upload). The path to the data in the datastore can be the root /, a directory within the datastore, or a file in the datastore.

The following example shows how to work with a DataReference object in an estimation pipeline.


   from azureml.core import Datastore

   def_blob_store = Datastore(ws, "workspaceblobstore")

   #upload input data to workspaceblobstore
   def_blob_store.upload_files(files=['20news.pkl'], target_path='20newsgroups')

Full sample is available from https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-how-to-use-estimatorstep.ipynb

Methods

as_download

Switch data reference operation to download.

DataReference download only supports Azure Blob and Azure File Share. To download data from Azure Blob, Azure File Share, Azure Data Lake Gen1, and Azure Data Lake Gen2 we recommend using Azure Machine Learning Dataset. For more information on how to create and use Dataset, please visit https://docs.microsoft.com/en-us/azure/machine-learning/how-to-train-with-datasets.

as_mount

Switch data reference operation to mount.

DataReference mount only supports Azure Blob. To mount data in Azure Blob, Azure File Share, Azure Data Lake Gen1, and Azure Data Lake Gen2 we recommend using Azure Machine Learning Dataset. For more information on how to create and use Dataset, please visit https://docs.microsoft.com/en-us/azure/machine-learning/how-to-train-with-datasets.

as_upload

Switch data reference operation to upload.

For more information on which computes and datastores support uploading of the data, see: https://aka.ms/datastore-matrix.

create

Create a DataReference using DataPath and DataPathComputeBinding.

path

Create a DataReference instance based on the given path.

to_config

Convert the DataReference object to DataReferenceConfiguration object.

as_download

Switch data reference operation to download.

DataReference download only supports Azure Blob and Azure File Share. To download data from Azure Blob, Azure File Share, Azure Data Lake Gen1, and Azure Data Lake Gen2 we recommend using Azure Machine Learning Dataset. For more information on how to create and use Dataset, please visit https://docs.microsoft.com/en-us/azure/machine-learning/how-to-train-with-datasets.

as_download(path_on_compute=None, overwrite=False)

Parameters

path_on_compute
str
default value: None

The path on the compute for the data reference.

overwrite
bool
default value: False

Indicates whether to overwrite existing data.

Returns

A new data reference object.

Return type

as_mount

Switch data reference operation to mount.

DataReference mount only supports Azure Blob. To mount data in Azure Blob, Azure File Share, Azure Data Lake Gen1, and Azure Data Lake Gen2 we recommend using Azure Machine Learning Dataset. For more information on how to create and use Dataset, please visit https://docs.microsoft.com/en-us/azure/machine-learning/how-to-train-with-datasets.

as_mount()

Returns

A new data reference object.

Return type

as_upload

Switch data reference operation to upload.

For more information on which computes and datastores support uploading of the data, see: https://aka.ms/datastore-matrix.

as_upload(path_on_compute=None, overwrite=False)

Parameters

path_on_compute
str
default value: None

The path on the compute for the data reference.

overwrite
bool
default value: False

Indicates whether to overwrite existing data.

Returns

A new data reference object.

Return type

create

Create a DataReference using DataPath and DataPathComputeBinding.

create(data_reference_name=None, datapath=None, datapath_compute_binding=None)

Parameters

data_reference_name
str
default value: None

The name for the data reference to create.

datapath
DataPath
default value: None

[Required] The datapath to use.

datapath_compute_binding
DataPathComputeBinding
default value: None

[Required] The datapath compute binding to use.

Returns

A DataReference object.

Return type

path

Create a DataReference instance based on the given path.

path(path=None, data_reference_name=None)

Parameters

path
str
default value: None

The path on the datastore.

data_reference_name
str
default value: None

The name of the data reference.

Returns

The data reference object.

Return type

to_config

Convert the DataReference object to DataReferenceConfiguration object.

to_config()

Returns

A new DataReferenceConfiguration object.

Return type