Copy data to or from a file system by using Azure Data Factory

This article outlines how to use the Copy Activity in Azure Data Factory to copy data from and to file system. It builds on the copy activity overview article that presents a general overview of copy activity.

Note

This article applies to version 2 of Data Factory, which is currently in preview. If you are using version 1 of the Data Factory service, which is generally available (GA), see File System connector in V1.

Supported capabilities

You can copy data from file system to any supported sink data store, or copy data from any supported source data store to file system. For a list of data stores that are supported as sources/sinks by the copy activity, see the Supported data stores table.

Specifically, this file system connector supports:

  • Copying files from/to local machine or network file share. To use a Linux file share, install Samba on your Linux server.
  • Copying files using Windows authentication.
  • Copying files as-is or parsing/generating files with the supported file formats and compression codecs.

Prerequisites

To copy data from/to a file system that is not publicly accessible, you need to set up a Self-hosted Integration Runtime. See Self-hosted Integration Runtime article for details.

Getting started

You can create a pipeline with the copy activity by using one of the following tools or SDKs. Select a link to go to a tutorial with step-by-step instructions to create a pipeline with a copy activity.

The following sections provide details about properties that are used to define Data Factory entities specific to file system.

Linked service properties

The following properties are supported for file system linked service:

Property Description Required
type The type property must be set to: FileServer. Yes
host Specifies the root path of the folder that you want to copy. Use the escape character "" for special characters in the string. See Sample linked service and dataset definitions for examples. Yes
userid Specify the ID of the user who has access to the server. Yes
password Specify the password for the user (userid). Mark this field as a SecureString to store it securely in Data Factory, or reference a secret stored in Azure Key Vault. Yes
connectVia The Integration Runtime to be used to connect to the data store. You can use Self-hosted Integration Runtime or Azure Integration Runtime (if your data store is publicly accessible). If not specified, it uses the default Azure Integration Runtime. No

Sample linked service and dataset definitions

Scenario "host" in linked service definition "folderPath" in dataset definition
Local folder on Integration Runtime machine:

Examples: D:\* or D:\folder\subfolder\*
D:\\ .\\ or folder\\subfolder
Remote shared folder:

Examples: \\myserver\share\* or \\myserver\share\folder\subfolder\*
\\\\myserver\\share .\\ or folder\\subfolder

Example:

{
    "name": "FileLinkedService",
    "properties": {
        "type": "FileServer",
        "typeProperties": {
            "host": "<host>",
            "userid": "<domain>\\<user>",
            "password": {
                "type": "SecureString",
                "value": "<password>"
            }
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

Dataset properties

For a full list of sections and properties available for defining datasets, see the datasets article. This section provides a list of properties supported by file system dataset.

To copy data from/to file system, set the type property of the dataset to FileShare. The following properties are supported:

Property Description Required
type The type property of the dataset must be set to: FileShare Yes
folderPath Path to the folder. See Sample linked service and dataset definitions for examples. Yes
fileName Specify the name of the file in the folderPath if you want to copy to/from a specific file. If you do not specify any value for this property, the dataset points to all files in the folder as source, and automatically generates file name.

fileName auto generation for sink: when fileName is not specified for an output dataset and preserveHierarchy is not specified in activity sink, copy activity generates the file name with the following pattern:
- Data_[activity run id]_[GUID].[format].[compression if configured]. For example: Data_0a405f8a-93ff-4c6f-b3be-f69616f1df7a_0d143eda-d5b8-44df-82ec-95c50895ff80.txt.gz
- or [Table name].[format].[compression if configured] for relational source when query is not specified. For example: MySourceTable.orc.
No
fileFilter Specify a filter to be used to select a subset of files in the folderPath rather than all files. Applies only when fileName is not specified.

Allowed wildcards are: * (multiple characters) and ? (single character).
- Example 1: "fileFilter": "*.log"
- Example 2: "fileFilter": 2017-09-??.txt"
No
format If you want to copy files as-is between file-based stores (binary copy), skip the format section in both input and output dataset definitions.

If you want to parse or generate files with a specific format, the following file format types are supported: TextFormat, JsonFormat, AvroFormat, OrcFormat, ParquetFormat. Set the type property under format to one of these values. For more information, see Text Format, Json Format, Avro Format, Orc Format, and Parquet Format sections.
No (only for binary copy scenario)
compression Specify the type and level of compression for the data. For more information, see Supported file formats and compression codecs.
Supported types are: GZip, Deflate, BZip2, and ZipDeflate.
Supported levels are: Optimal and Fastest.
No

Example:

{
    "name": "FileSystemDataset",
    "properties": {
        "type": "FileShare",
        "linkedServiceName":{
            "referenceName": "<file system linked service name>",
            "type": "LinkedServiceReference"
        },
        "typeProperties": {
            "folderPath": "folder/subfolder/",
            "fileName": "myfile.csv.gz",
            "format": {
                "type": "TextFormat",
                "columnDelimiter": ",",
                "rowDelimiter": "\n"
            },
            "compression": {
                "type": "GZip",
                "level": "Optimal"
            }
        }
    }
}

Copy activity properties

For a full list of sections and properties available for defining activities, see the Pipelines article. This section provides a list of properties supported by file system source and sink.

File system as source

To copy data from file system, set the source type in the copy activity to FileSystemSource. The following properties are supported in the copy activity source section:

Property Description Required
type The type property of the copy activity source must be set to: FileSystemSource Yes
recursive Indicates whether the data is read recursively from the sub-folders or only from the specified folder. Note when recursive is set to true and sink is file-based store, empty folder/sub-folder will not be copied/created at sink.
Allowed values are: true (default), false
No

Example:

"activities":[
    {
        "name": "CopyFromFileSystem",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<file system input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "FileSystemSource",
                "recursive": true
            },
            "sink": {
                "type": "<sink type>"
            }
        }
    }
]

File system as sink

To copy data to file system, set the sink type in the copy activity to FileSystemSink. The following properties are supported in the sink section:

Property Description Required
type The type property of the copy activity sink must be set to: FileSystemSink Yes
copyBehavior Defines the copy behavior when the source is files from file-based data store.

Allowed values are:
- PreserveHierarchy (default): preserves the file hierarchy in the target folder. The relative path of source file to source folder is identical to the relative path of target file to target folder.
- FlattenHierarchy: all files from the source folder are in the first level of target folder. The target files have auto generated name.
- MergeFiles: merges all files from the source folder to one file. If the File/Blob Name is specified, the merged file name would be the specified name; otherwise, would be auto-generated file name.
No

Example:

"activities":[
    {
        "name": "CopyToFileSystem",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<file system output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "<source type>"
            },
            "sink": {
                "type": "FileSystemSink",
                "copyBehavior": "PreserveHierarchy"
            }
        }
    }
]

recursive and copyBehavior examples

This section describes the resulting behavior of the Copy operation for different combinations of recursive and copyBehavior values.

recursive copyBehavior Source folder structure Resulting target
true preserveHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target folder Folder1 is created with the same structure as the source:

Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5.
true flattenHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target Folder1 is created with the following structure:

Folder1
    auto-generated name for File1
    auto-generated name for File2
    auto-generated name for File3
    auto-generated name for File4
    auto-generated name for File5
true mergeFiles Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target Folder1 is created with the following structure:

Folder1
    File1 + File2 + File3 + File4 + File 5 contents are merged into one file with auto-generated file name
false preserveHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target folder Folder1 is created with the following structure

Folder1
    File1
    File2

Subfolder1 with File3, File4, and File5 are not picked up.
false flattenHierarchy Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target folder Folder1 is created with the following structure

Folder1
    auto-generated name for File1
    auto-generated name for File2

Subfolder1 with File3, File4, and File5 are not picked up.
false mergeFiles Folder1
    File1
    File2
    Subfolder1
        File3
        File4
        File5
The target folder Folder1 is created with the following structure

Folder1
    File1 + File2 contents are merged into one file with auto-generated file name. auto-generated name for File1

Subfolder1 with File3, File4, and File5 are not picked up.

Next steps

For a list of data stores supported as sources and sinks by the copy activity in Azure Data Factory, see supported data stores.