Copy data to or from Azure Data Lake Storage Gen1 by using Azure Data Factory

This article outlines how to use Copy Activity in Azure Data Factory to copy data to and from Azure Data Lake Storage Gen1 (previously known as Azure Data Lake Store). It builds on the Copy Activity overview.

Supported capabilities

You can copy data from any supported source data store to Azure Data Lake Store, or copy data from Azure Data Lake Store to any supported sink data store. See the supported data stores table.

Specifically, this Azure Data Lake Store connector supports:

  • Copying files by using one of the following methods of authentication: service principal or managed identities for Azure resources.
  • Copying files as-is, or parsing or generating files with the supported file formats and compression codecs.

Important

If you copy data using the self-hosted integration runtime, configure the corporate firewall to allow outbound traffic to <ADLS account name>.azuredatalakestore.net and login.microsoftonline.com/<tenant>/oauth2/token on port 443. The latter is the Azure Security Token Service that the integration runtime needs to communicate with to get the access token.

Get started

Tip

For a walkthrough of using the Azure Data Lake Store connector, see Load data into Azure Data Lake Store.

You can use one of the following tools or SDKs to use Copy Activity with a pipeline. Select a link for step-by-step instructions:

The following sections provide details about properties that are used to define Data Factory entities specific to Azure Data Lake Store.

Linked service properties

The following properties are supported for the Azure Data Lake Store linked service:

Property Description Required
type The type property must be set to AzureDataLakeStore. Yes
dataLakeStoreUri Information about the Azure Data Lake Store account. This information takes one of the following formats: https://[accountname].azuredatalakestore.net/webhdfs/v1 or adl://[accountname].azuredatalakestore.net/. Yes
subscriptionId The Azure subscription ID to which the Data Lake Store account belongs. Required for sink
resourceGroupName The Azure resource group name to which the Data Lake Store account belongs. Required for sink
connectVia The integration runtime to be used to connect to the data store. You can use Azure integration runtime or self-hosted integration runtime (if your data store is located in a private network). If this property is not specified, it uses the default Azure integration runtime. No

Use service principal authentication

To use service principal authentication, register an application entity in Azure Active Directory and grant it access to Data Lake Store. For detailed steps, see Service-to-service authentication. Make note of the following values, which you use to define the linked service:

  • Application ID
  • Application key
  • Tenant ID

Important

Make sure you grant the service principal proper permission in Data Lake Store:

  • As source: In Data explorer > Access, grant at least Read + Execute permission to list and copy the files in folders and subfolders. Or, you can grant Read permission to copy a single file. You can choose to add to This folder and all children for recursive, and add as an access permission and a default permission entry. There's no requirement on account level access control (IAM).
  • As sink: In Data explorer > Access, grant at least Write + Execute permission to create child items in the folder. You can choose to add to This folder and all children for recursive, and add as an access permission and a default permission entry. If you use Azure integration runtime to copy (both source and sink are in the cloud), in IAM, grant at least the Reader role in order to let Data Factory detect the region for Data Lake Store. If you want to avoid this IAM role, explicitly create an Azure integration runtime with the location of Data Lake Store. Associate them in the Data Lake Store linked service as the following example.

Note

To list folders starting from the root, you must set the permission of the service principal being granted to at root level with "Execute" permission. This is true when you use the:

  • Copy Data Tool to author copy pipeline.
  • Data Factory UI to test connection and navigating folders during authoring.

The following properties are supported:

Property Description Required
servicePrincipalId Specify the application's client ID. Yes
servicePrincipalKey Specify the application's key. Mark this field as a SecureString to store it securely in Data Factory, or reference a secret stored in Azure Key Vault. Yes
tenant Specify the tenant information (domain name or tenant ID) under which your application resides. You can retrieve it by hovering the mouse in the upper-right corner of the Azure portal. Yes

Example:

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://<accountname>.azuredatalakestore.net/webhdfs/v1",
            "servicePrincipalId": "<service principal id>",
            "servicePrincipalKey": {
                "type": "SecureString",
                "value": "<service principal key>"
            },
            "tenant": "<tenant info, e.g. microsoft.onmicrosoft.com>",
            "subscriptionId": "<subscription of ADLS>",
            "resourceGroupName": "<resource group of ADLS>"
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

Use managed identities for Azure resources authentication

A data factory can be associated with a managed identity for Azure resources, which represents this specific data factory. You can directly use this service identity for Data Lake Store authentication, similar to using your own service principal. It allows this designated factory to access and copy data to or from Data Lake Store.

To use managed identities for Azure resources authentication:

  1. Retrieve the data factory service identity by copying the value of the "Service Identity Application ID" generated along with your factory.
  2. Grant the service identity access to Data Lake Store, the same way you do for service principal, following these notes.

Important

Make sure you grant the data factory service identity proper permission in Data Lake Store:

  • As source: In Data explorer > Access, grant at least Read + Execute permission to list and copy the files in folders and subfolders. Or, you can grant Read permission to copy a single file. You can choose to add to This folder and all children for recursive, and add as an access permission and a default permission entry. There's no requirement on account level access control (IAM).
  • As sink: In Data explorer > Access, grant at least Write + Execute permission to create child items in the folder. You can choose to add to This folder and all children for recursive, and add as an access permission and a default permission entry. If you use Azure integration runtime to copy (both source and sink are in the cloud), in IAM, grant at least the Reader role in order to let Data Factory detect the region for Data Lake Store. If you want to avoid this IAM role, explicitly create an Azure integration runtime with the location of Data Lake Store. Associate them in the Data Lake Store linked service as the following example.

Note

To list folders starting from the root, you must set the permission of the service principal being granted to at root level with "Execute" permission. This is true when you use the:

  • Copy Data Tool to author copy pipeline.
  • Data Factory UI to test connection and navigating folders during authoring.

In Azure Data Factory, you don't need to specify any properties besides the general Data Lake Store information in the linked service.

Example:

{
    "name": "AzureDataLakeStoreLinkedService",
    "properties": {
        "type": "AzureDataLakeStore",
        "typeProperties": {
            "dataLakeStoreUri": "https://<accountname>.azuredatalakestore.net/webhdfs/v1",
            "subscriptionId": "<subscription of ADLS>",
            "resourceGroupName": "<resource group of ADLS>"
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

Dataset properties

To copy data to and from Azure Data Lake Store, set the type property of the dataset to AzureDataLakeStoreFile. The following properties are supported:

Property Description Required
type The type property of the dataset must be set to: AzureDataLakeStoreFile Yes
folderPath Path to the folder in Data Lake Store. Wildcard filter is not supported. If not specified, it points to the root. Example: rootfolder/subfolder/ No
fileName Name or wildcard filter for the file(s) under the specified "folderPath". If you don't specify a value for this property, the dataset points to all files in the folder.

For filter, allowed wildcards are: * (matches zero or more characters) and ? (matches zero or single character).
- Example 1: "fileName": "*.csv"
- Example 2: "fileName": "???20180427.txt"
Use ^ to escape if your actual file name has wildcard or this escape char inside.

When fileName isn't specified for an output dataset and preserveHierarchy isn't specified in the activity sink, the copy activity automatically generates the file name with the following pattern: "Data.[activity run id GUID].[GUID if FlattenHierarchy].[format if configured].[compression if configured]". For example, "Data.0a405f8a-93ff-4c6f-b3be-f69616f1df7a.txt.gz". If you copy from tabular source using table name instead of query, the name pattern is "[table name].[format].[compression if configured]". For example, "MyTable.csv".
No
format If you want to copy files as-is between file-based stores (binary copy), skip the format section in both input and output dataset definitions.

If you want to parse or generate files with a specific format, the following file format types are supported: TextFormat, JsonFormat, AvroFormat, OrcFormat, ParquetFormat. Set the type property under format to one of these values. For more information, see Text Format, Json Format, Avro Format, Orc Format, and Parquet Format sections.
No (only for binary copy scenario)
compression Specify the type and level of compression for the data. For more information, see Supported file formats and compression codecs.
Supported types are: GZip, Deflate, BZip2, and ZipDeflate.
Supported levels are: Optimal and Fastest.
No

Tip

To copy all files under a folder, specify folderPath only.
To copy a single file with a particular name, specify folderPath with folder part and fileName with file name.
To copy a subset of files under a folder, specify folderPath with folder part and fileName with wildcard filter.

Example:

{
    "name": "ADLSDataset",
    "properties": {
        "type": "AzureDataLakeStoreFile",
        "linkedServiceName":{
            "referenceName": "<ADLS linked service name>",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "folderPath": "datalake/myfolder/",
            "fileName": "myfile.csv.gz",
            "format": {
                "type": "TextFormat",
                "columnDelimiter": ",",
                "rowDelimiter": "\n"
            },
            "compression": {
                "type": "GZip",
                "level": "Optimal"
            }
        }
    }
}

Copy Activity properties

For a full list of sections and properties available for defining activities, see Pipelines. This section provides a list of properties supported by Azure Data Lake Store source and sink.

Azure Data Lake Store as source

To copy data from Data Lake Store, set the source type in Copy Activity to AzureDataLakeStoreSource. The following properties are supported in the Copy Activity source section:

Property Description Required
type The type property of the Copy Activity source must be set to: AzureDataLakeStoreSource. Yes
recursive Indicates whether the data is read recursively from the subfolders or only from the specified folder. Note that when recursive is set to true and the sink is a file-based store, an empty folder or subfolder isn't copied or created at the sink. Allowed values are: true (default) and false. No

Example:

"activities":[
    {
        "name": "CopyFromADLS",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<ADLS input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "AzureDataLakeStoreSource",
                "recursive": true
            },
            "sink": {
                "type": "<sink type>"
            }
        }
    }
]

Azure Data Lake Store as sink

To copy data to Data Lake Store, set the sink type in Copy Activity to AzureDataLakeStoreSink. The following properties are supported in the sink section:

Property Description Required
type The type property of the Copy Activity sink must be set to: AzureDataLakeStoreSink. Yes
copyBehavior Defines the copy behavior when the source is files from a file-based data store.

Allowed values are:
- PreserveHierarchy (default): preserves the file hierarchy in the target folder. The relative path of the source file to the source folder is identical to the relative path of the target file to the target folder.
- FlattenHierarchy: all files from the source folder are in the first level of the target folder. The target files have auto-generated names.
- MergeFiles: merges all files from the source folder to one file. If the file/blob name is specified, the merged file name is the specified name. Otherwise, the file name is auto-generated.
No

Example:

"activities":[
    {
        "name": "CopyToADLS",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<ADLS output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "<source type>"
            },
            "sink": {
                "type": "AzureDataLakeStoreSink",
                "copyBehavior": "PreserveHierarchy"
            }
        }
    }
]

Examples of behavior of the copy operation

This section describes the resulting behavior of the copy operation for different combinations of recursive and copyBehavior values.

recursive copyBehavior Source folder structure Resulting target
true preserveHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target Folder1 is created with the same structure as the source:

Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5.
true flattenHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target Folder1 is created with the following structure:

Folder1
    auto-generated name for File1
    auto-generated name for File2
    auto-generated name for File3
    auto-generated name for File4
    auto-generated name for File5
true mergeFiles Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target Folder1 is created with the following structure:

Folder1
    File1 + File2 + File3 + File4 + File 5 contents are merged into one file, with an auto-generated file name.
false preserveHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target Folder1 is created with the following structure:

Folder1
    File1
    File2

Subfolder1 with File3, File4, and File5 are not picked up.
false flattenHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target Folder1 is created with the following structure:

Folder1
    auto-generated name for File1
    auto-generated name for File2

Subfolder1 with File3, File4, and File5 are not picked up.
false mergeFiles Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target Folder1 is created with the following structure:

Folder1
    File1 + File2 contents are merged into one file with auto-generated file name. auto-generated name for File1

Subfolder1 with File3, File4, and File5 are not picked up.

Next steps

For a list of data stores supported as sources and sinks by Copy Activity in Azure Data Factory, see supported data stores.