Avro format in Azure Data Factory

APPLIES TO: yesAzure Data Factory yesAzure Synapse Analytics (Preview)

Follow this article when you want to parse the Avro files or write the data into Avro format.

Avro format is supported for the following connectors: Amazon S3, Azure Blob, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, Azure File Storage, File System, FTP, Google Cloud Storage, HDFS, HTTP, and SFTP.

Dataset properties

For a full list of sections and properties available for defining datasets, see the Datasets article. This section provides a list of properties supported by the Avro dataset.

Property Description Required
type The type property of the dataset must be set to Avro. Yes
location Location settings of the file(s). Each file-based connector has its own location type and supported properties under location. See details in connector article -> Dataset properties section. Yes
avroCompressionCodec The compression codec to use when writing to Avro files. When reading from Avro files, Data Factory automatically determine the compression codec based on the file metadata.
Supported types are "none" (default), "deflate", "snappy". Note currently Copy activity doesn't support Snappy when read/write Avro files.
No

Note

White space in column name is not supported for Avro files.

Below is an example of Avro dataset on Azure Blob Storage:

{
    "name": "AvroDataset",
    "properties": {
        "type": "Avro",
        "linkedServiceName": {
            "referenceName": "<Azure Blob Storage linked service name>",
            "type": "LinkedServiceReference"
        },
        "schema": [ < physical schema, optional, retrievable during authoring > ],
        "typeProperties": {
            "location": {
                "type": "AzureBlobStorageLocation",
                "container": "containername",
                "folderPath": "folder/subfolder",
            },
            "avroCompressionCodec": "snappy"
        }
    }
}

Copy activity properties

For a full list of sections and properties available for defining activities, see the Pipelines article. This section provides a list of properties supported by the Avro source and sink.

Avro as source

The following properties are supported in the copy activity *source* section.

Property Description Required
type The type property of the copy activity source must be set to AvroSource. Yes
storeSettings A group of properties on how to read data from a data store. Each file-based connector has its own supported read settings under storeSettings. See details in connector article -> Copy activity properties section. No

Avro as sink

The following properties are supported in the copy activity *sink* section.

Property Description Required
type The type property of the copy activity source must be set to AvroSink. Yes
storeSettings A group of properties on how to write data to a data store. Each file-based connector has its own supported write settings under storeSettings. See details in connector article -> Copy activity properties section. No

Mapping data flow properties

In mapping data flows, you can read and write to avro format in the following data stores: Azure Blob Storage, Azure Data Lake Storage Gen1, and Azure Data Lake Storage Gen2.

Source properties

The below table lists the properties supported by a avro source. You can edit these properties in the Source options tab.

Name Description Required Allowed values Data flow script property
Wild card paths All files matching the wildcard path will be processed. Overrides the folder and file path set in the dataset. no String[] wildcardPaths
Partition root path For file data that is partitioned, you can enter a partition root path in order to read partitioned folders as columns no String partitionRootPath
List of files Whether your source is pointing to a text file that lists files to process no true or false fileList
Column to store file name Create a new column with the source file name and path no String rowUrlColumn
After completion Delete or move the files after processing. File path starts from the container root no Delete: true or false
Move: ['<from>', '<to>']
purgeFiles
moveFiles
Filter by last modified Choose to filter files based upon when they were last altered no Timestamp modifiedAfter
modifiedBefore

Sink properties

The below table lists the properties supported by a avro sink. You can edit these properties in the Settings tab.

Name Description Required Allowed values Data flow script property
Clear the folder If the destination folder is cleared prior to write no true or false truncate
File name option The naming format of the data written. By default, one file per partition in format part-#####-tid-<guid> no Pattern: String
Per partition: String[]
As data in column: String
Output to single file: ['<fileName>']
filePattern
partitionFileNames
rowUrlColumn
partitionFileNames
Quote all Enclose all values in quotes no true or false quoteAll

Data type support

Copy activity

Avro complex data types are not supported (records, enums, arrays, maps, unions, and fixed) in Copy Activity.

Data flows

When working with Avro files in data flows, you can read and write complex data types, but be sure to clear the physical schema from the dataset first. In data flows, you can set your logical projection and derive columns that are complex structures, then auto-map those fields to an Avro file.

Next steps