Transfer data with AzCopy on Linux

AzCopy on Linux is a command-line utility designed for copying data to and from Microsoft Azure Blob and File storage using simple commands with optimal performance. You can copy data from one object to another within your storage account, or between storage accounts.

There are two versions of AzCopy that you can download. AzCopy on Linux is built with .NET Core Framework, which targets Linux platforms offering POSIX style command-line options. AzCopy on Windows is built with .NET Framework, and offers Windows style command-line options. This article covers AzCopy on Linux.

Download and install AzCopy

Installation on Linux

The article includes commands for various releases of Ubuntu. Use the lsb_release -a command to confirm your distribution release and codename.

AzCopy on Linux requires .NET Core framework (version 1.1.x) on the platform. See the installation instructions on the .NET Core page.

As an example, let's install .NET Core on Ubuntu 16.04. For the latest installation guide, visit .NET Core on Linux installation page.

curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/dotnetdev.list'
sudo apt-get update
sudo apt-get install dotnet-dev-1.1.4

Once you have installed .NET Core, download and install AzCopy.

wget -O azcopy.tar.gz https://aka.ms/downloadazcopyprlinux
tar -xf azcopy.tar.gz
sudo ./install.sh

You can remove the extracted files once AzCopy on Linux is installed. Alternatively if you do not have superuser privileges, you can also run AzCopy using the shell script 'azcopy' in the extracted folder.

Writing your first AzCopy command

The basic syntax for AzCopy commands is:

azcopy --source <source> --destination <destination> [Options]

The following examples demonstrate various scenarios for copying data to and from Microsoft Azure Blobs and Files. Refer to the azcopy --help menu for a detailed explanation of the parameters used in each sample.

Blob: Download

Download single blob

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer \
    --destination /mnt/myfiles \
    --source-key <key> \
    --include "abc.txt"

If the folder /mnt/myfiles does not exist, AzCopy creates it and downloads abc.txt into the new folder.

Download single blob from secondary region

azcopy \
    --source https://myaccount-secondary.blob.core.windows.net/mynewcontainer \
    --destination /mnt/myfiles \
    --source-key <key> \
    --include "abc.txt"

Note that you must have read-access geo-redundant storage enabled.

Download all blobs

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer \
    --destination /mnt/myfiles \
    --source-key <key> \
    --recursive

Assume the following blobs reside in the specified container:

abc.txt
abc1.txt
abc2.txt
vd1/a.txt
vd1/abcd.txt

After the download operation, the directory /mnt/myfiles includes the following files:

/mnt/myfiles/abc.txt
/mnt/myfiles/abc1.txt
/mnt/myfiles/abc2.txt
/mnt/myfiles/vd1/a.txt
/mnt/myfiles/vd1/abcd.txt

If you do not specify option --recursive, no blob will be downloaded.

Download blobs with specified prefix

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer \
    --destination /mnt/myfiles \
    --source-key <key> \
    --include "a" \
    --recursive

Assume the following blobs reside in the specified container. All blobs beginning with the prefix a are downloaded.

abc.txt
abc1.txt
abc2.txt
xyz.txt
vd1\a.txt
vd1\abcd.txt

After the download operation, the folder /mnt/myfiles includes the following files:

/mnt/myfiles/abc.txt
/mnt/myfiles/abc1.txt
/mnt/myfiles/abc2.txt

The prefix applies to the virtual directory, which forms the first part of the blob name. In the example shown above, the virtual directory does not match the specified prefix, so no blob is downloaded. In addition, if the option --recursive is not specified, AzCopy does not download any blobs.

Set the last-modified time of exported files to be same as the source blobs

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer \
    --destination "/mnt/myfiles" \
    --source-key <key> \
    --preserve-last-modified-time

You can also exclude blobs from the download operation based on their last-modified time. For example, if you want to exclude blobs whose last modified time is the same or newer than the destination file, add the --exclude-newer option:

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer \
    --destination /mnt/myfiles \
    --source-key <key> \
    --preserve-last-modified-time \
    --exclude-newer

Or if you want to exclude blobs whose last modified time is the same or older than the destination file, add the --exclude-older option:

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer \
    --destination /mnt/myfiles \
    --source-key <key> \
    --preserve-last-modified-time \
    --exclude-older

Blob: Upload

Upload single file

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/mycontainer \
    --dest-key <key> \
    --include "abc.txt"

If the specified destination container does not exist, AzCopy creates it and uploads the file into it.

Upload single file to virtual directory

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/mycontainer \
    --dest-key <key> \
    --include "abc.txt"

If the specified virtual directory does not exist, AzCopy uploads the file to include the virtual directory in the blob name (e.g., vd/abc.txt in the example above).

Upload all files

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/mycontainer \
    --dest-key <key> \
    --recursive

Specifying option --recursive uploads the contents of the specified directory to Blob storage recursively, meaning that all subfolders and their files are uploaded as well. For instance, assume the following files reside in folder /mnt/myfiles:

/mnt/myfiles/abc.txt
/mnt/myfiles/abc1.txt
/mnt/myfiles/abc2.txt
/mnt/myfiles/subfolder/a.txt
/mnt/myfiles/subfolder/abcd.txt

After the upload operation, the container includes the following files:

abc.txt
abc1.txt
abc2.txt
subfolder/a.txt
subfolder/abcd.txt

When the option --recursive is not specified, only the following three files are uploaded:

abc.txt
abc1.txt
abc2.txt

Upload files matching specified pattern

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/mycontainer \
    --dest-key <key> \
    --include "a*" \
    --recursive

Assume the following files reside in folder /mnt/myfiles:

/mnt/myfiles/abc.txt
/mnt/myfiles/abc1.txt
/mnt/myfiles/abc2.txt
/mnt/myfiles/xyz.txt
/mnt/myfiles/subfolder/a.txt
/mnt/myfiles/subfolder/abcd.txt

After the upload operation, the container includes the following files:

abc.txt
abc1.txt
abc2.txt
subfolder/a.txt
subfolder/abcd.txt

When the option --recursive is not specified, AzCopy skips files that are in sub-directories:

abc.txt
abc1.txt
abc2.txt

Specify the MIME content type of a destination blob

By default, AzCopy sets the content type of a destination blob to application/octet-stream. However, you can explicitly specify the content type via the option --set-content-type [content-type]. This syntax sets the content type for all blobs in an upload operation.

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/myContainer/ \
    --dest-key <key> \
    --include "ab" \
    --set-content-type "video/mp4"

If the option --set-content-type is specified without a value, then AzCopy sets each blob or file's content type according to its file extension.

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/myContainer/ \
    --dest-key <key> \
    --include "ab" \
    --set-content-type

Blob: Copy

Copy single blob within Storage account

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer1 \
    --destination https://myaccount.blob.core.windows.net/mycontainer2 \
    --source-key <key> \
    --dest-key <key> \
    --include "abc.txt"

When you copy a blob without --sync-copy option, a server-side copy operation is performed.

Copy single blob across Storage accounts

azcopy \
    --source https://sourceaccount.blob.core.windows.net/mycontainer1 \
    --destination https://destaccount.blob.core.windows.net/mycontainer2 \
    --source-key <key1> \
    --dest-key <key2> \
    --include "abc.txt"

When you copy a blob without --sync-copy option, a server-side copy operation is performed.

Copy single blob from secondary region to primary region

azcopy \
    --source https://myaccount1-secondary.blob.core.windows.net/mynewcontainer1 \
    --destination https://myaccount2.blob.core.windows.net/mynewcontainer2 \
    --source-key <key1> \
    --dest-key <key2> \
    --include "abc.txt"

Note that you must have read-access geo-redundant storage enabled.

Copy single blob and its snapshots across Storage accounts

azcopy \
    --source https://sourceaccount.blob.core.windows.net/mycontainer1 \
    --destination https://destaccount.blob.core.windows.net/mycontainer2 \
    --source-key <key1> \
    --dest-key <key2> \
    --include "abc.txt" \
    --include-snapshot

After the copy operation, the target container includes the blob and its snapshots. The container includes the following blob and its snapshots:

abc.txt
abc (2013-02-25 080757).txt
abc (2014-02-21 150331).txt

Synchronously copy blobs across Storage accounts

AzCopy by default copies data between two storage endpoints asynchronously. Therefore, the copy operation runs in the background using spare bandwidth capacity that has no SLA in terms of how fast a blob is copied.

The --sync-copy option ensures that the copy operation gets consistent speed. AzCopy performs the synchronous copy by downloading the blobs to copy from the specified source to local memory, and then uploading them to the Blob storage destination.

azcopy \
    --source https://myaccount1.blob.core.windows.net/myContainer/ \
    --destination https://myaccount2.blob.core.windows.net/myContainer/ \
    --source-key <key1> \
    --dest-key <key2> \
    --include "ab" \
    --sync-copy

--sync-copy might generate additional egress cost compared to asynchronous copy. The recommended approach is to use this option in an Azure VM, that is in the same region as your source storage account to avoid egress cost.

File: Download

Download single file

azcopy \
    --source https://myaccount.file.core.windows.net/myfileshare/myfolder1/ \
    --destination /mnt/myfiles \
    --source-key <key> \
    --include "abc.txt"

If the specified source is an Azure file share, then you must either specify the exact file name, (e.g. abc.txt) to download a single file, or specify option --recursive to download all files in the share recursively. Attempting to specify both a file pattern and option --recursive together results in an error.

Download all files

azcopy \
    --source https://myaccount.file.core.windows.net/myfileshare/ \
    --destination /mnt/myfiles \
    --source-key <key> \
    --recursive

Note that any empty folders are not downloaded.

File: Upload

Upload single file

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.file.core.windows.net/myfileshare/ \
    --dest-key <key> \
    --include abc.txt

Upload all files

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.file.core.windows.net/myfileshare/ \
    --dest-key <key> \
    --recursive

Note that any empty folders are not uploaded.

Upload files matching specified pattern

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.file.core.windows.net/myfileshare/ \
    --dest-key <key> \
    --include "ab*" \
    --recursive

File: Copy

Copy across file shares

azcopy \
    --source https://myaccount1.file.core.windows.net/myfileshare1/ \
    --destination https://myaccount2.file.core.windows.net/myfileshare2/ \
    --source-key <key1> \
    --dest-key <key2> \
    --recursive

When you copy a file across file shares, a server-side copy operation is performed.

Copy from file share to blob

azcopy \ 
    --source https://myaccount1.file.core.windows.net/myfileshare/ \
    --destination https://myaccount2.blob.core.windows.net/mycontainer/ \
    --source-key <key1> \
    --dest-key <key2> \
    --recursive

When you copy a file from file share to blob, a server-side copy operation is performed.

Copy from blob to file share

azcopy \
    --source https://myaccount1.blob.core.windows.net/mycontainer/ \
    --destination https://myaccount2.file.core.windows.net/myfileshare/ \
    --source-key <key1> \
    --dest-key <key2> \
    --recursive

When you copy a file from blob to file share, a server-side copy operation is performed.

Synchronously copy files

You can specify the --sync-copy option to copy data from File Storage to File Storage, from File Storage to Blob Storage and from Blob Storage to File Storage synchronously. AzCopy runs this operation by downloading the source data to local memory, and then uploading it to destination. In this case, standard egress cost applies.

azcopy \
    --source https://myaccount1.file.core.windows.net/myfileshare1/ \
    --destination https://myaccount2.file.core.windows.net/myfileshare2/ \
    --source-key <key1> \
    --dest-key <key2> \
    --recursive \
    --sync-copy

When copying from File Storage to Blob Storage, the default blob type is block blob, user can specify option --blob-type page to change the destination blob type. Available types are page | block | append.

Note that --sync-copy might generate additional egress cost comparing to asynchronous copy. The recommended approach is to use this option in an Azure VM, that is in the same region as your source storage account to avoid egress cost.

Other AzCopy features

Only copy data that doesn't exist in the destination

The --exclude-older and --exclude-newer parameters allow you to exclude older or newer source resources from being copied, respectively. If you only want to copy source resources that don't exist in the destination, you can specify both parameters in the AzCopy command:

--source http://myaccount.blob.core.windows.net/mycontainer --destination /mnt/myfiles --source-key <sourcekey> --recursive --exclude-older --exclude-newer

--source /mnt/myfiles --destination http://myaccount.file.core.windows.net/myfileshare --dest-key <destkey> --recursive --exclude-older --exclude-newer

--source http://myaccount.blob.core.windows.net/mycontainer --destination http://myaccount.blob.core.windows.net/mycontainer1 --source-key <sourcekey> --dest-key <destkey> --recursive --exclude-older --exclude-newer

Use a configuration file to specify command-line parameters

azcopy --config-file "azcopy-config.ini"

You can include any AzCopy command-line parameters in a configuration file. AzCopy processes the parameters in the file as if they had been specified on the command line, performing a direct substitution with the contents of the file.

Assume a configuration file named copyoperation, that contains the following lines. Each AzCopy parameter can be specified on a single line.

--source http://myaccount.blob.core.windows.net/mycontainer --destination /mnt/myfiles --source-key <sourcekey> --recursive --quiet

or on separate lines:

--source http://myaccount.blob.core.windows.net/mycontainer
--destination /mnt/myfiles
--source-key<sourcekey>
--recursive
--quiet

AzCopy fails if you split the parameter across two lines, as shown here for the --source-key parameter:

http://myaccount.blob.core.windows.net/mycontainer
/mnt/myfiles
--sourcekey
<sourcekey>
--recursive
--quiet

Specify a shared access signature (SAS)

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer1 \
    --destination https://myaccount.blob.core.windows.net/mycontainer2 \
    --source-sas <SAS1> \
    --dest-sas <SAS2> \
    --include abc.txt

You can also specify a SAS on the container URI:

azcopy \
    --source https://myaccount.blob.core.windows.net/mycontainer1/?SourceSASToken \
    --destination /mnt/myfiles \
    --recursive

Note that AzCopy currently only supports the Account SAS.

Journal file folder

Each time you issue a command to AzCopy, it checks whether a journal file exists in the default folder, or whether it exists in a folder that you specified via this option. If the journal file does not exist in either place, AzCopy treats the operation as new and generates a new journal file.

If the journal file does exist, AzCopy checks whether the command line that you input matches the command line in the journal file. If the two command lines match, AzCopy resumes the incomplete operation. If they do not match, AzCopy prompts user to either overwrite the journal file to start a new operation, or to cancel the current operation.

If you want to use the default location for the journal file:

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/mycontainer \
    --dest-key <key> \
    --resume

If you omit option --resume, or specify option --resume without the folder path, as shown above, AzCopy creates the journal file in the default location, which is ~\Microsoft\Azure\AzCopy. If the journal file already exists, then AzCopy resumes the operation based on the journal file.

If you want to specify a custom location for the journal file:

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/mycontainer \
    --dest-key key \
    --resume "/mnt/myjournal"

This example creates the journal file if it does not already exist. If it does exist, then AzCopy resumes the operation based on the journal file.

If you want to resume an AzCopy operation, repeat the same command. AzCopy on Linux then will prompt for confirmation:

Incomplete operation with same command line detected at the journal directory "/home/myaccount/Microsoft/Azure/AzCopy", do you want to resume the operation? Choose Yes to resume, choose No to overwrite the journal to start a new operation. (Yes/No)

Output verbose logs

azcopy \
    --source /mnt/myfiles \
    --destination https://myaccount.blob.core.windows.net/mycontainer \
    --dest-key <key> \
    --verbose

Specify the number of concurrent operations to start

Option --parallel-level specifies the number of concurrent copy operations. By default, AzCopy starts a certain number of concurrent operations to increase the data transfer throughput. The number of concurrent operations is equal eight times the number of processors you have. If you are running AzCopy across a low-bandwidth network, you can specify a lower number for --parallel-level to avoid failure caused by resource competition.

[!TIP]

To view the complete list of AzCopy parameters, check out 'azcopy --help' menu.

Known issues and best practices

Error: .NET Core is not found in the system.

If you encounter an error stating that .NET Core is not installed in the system, the PATH to the .NET Core binary dotnet may be missing.

In order to address this issue, find the .NET Core binary in the system:

sudo find / -name dotnet

This returns the path to the dotnet binary.

/opt/rh/rh-dotnetcore11/root/usr/bin/dotnet
/opt/rh/rh-dotnetcore11/root/usr/lib64/dotnetcore/dotnet
/opt/rh/rh-dotnetcore11/root/usr/lib64/dotnetcore/shared/Microsoft.NETCore.App/1.1.2/dotnet

Now add this path to the PATH variable. For sudo, edit secure_path to contain the path to the dotnet binary:

sudo visudo
### Append the path found in the preceding example to 'secure_path' variable

In this example, secure_path variable reads as:

secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/opt/rh/rh-dotnetcore11/root/usr/bin/

For the current user, edit .bash_profile/.profile to include the path to the dotnet binary in PATH variable

vi ~/.bash_profile
### Append the path found in the preceding example to 'PATH' variable

Verify that .NET Core is now in PATH:

which dotnet
sudo which dotnet

Error Installing AzCopy

If you encounter issues with AzCopy installation, you may try to run AzCopy using the bash script in the extracted azcopy folder.

cd azcopy
./azcopy

Limit concurrent writes while copying data

When you copy blobs or files with AzCopy, keep in mind that another application may be modifying the data while you are copying it. If possible, ensure that the data you are copying is not being modified during the copy operation. For example, when copying a VHD associated with an Azure virtual machine, make sure that no other applications are currently writing to the VHD. A good way to do this is by leasing the resource to be copied. Alternately, you can create a snapshot of the VHD first and then copy the snapshot.

If you cannot prevent other applications from writing to blobs or files while they are being copied, then keep in mind that by the time the job finishes, the copied resources may no longer have full parity with the source resources.

Run one AzCopy instance on one machine.

AzCopy is designed to maximize the utilization of your machine resource to accelerate the data transfer, we recommend you run only one AzCopy instance on one machine, and specify the option --parallel-level if you need more concurrent operations. For more details, type AzCopy --help parallel-level at the command line.

Next steps

For more information about Azure Storage and AzCopy, see the following resources:

Azure Storage documentation:

Azure Storage blog posts: