Tutorial: Copy data to Azure Data Box Blob storage via REST APIs

This tutorial describes procedures to connect to Azure Data Box Blob storage via REST APIs over http or https. Once connected, the steps required to copy the data to Data Box Blob storage and prepare the Data Box to ship, are also described.

In this tutorial, you learn how to:

  • Connect to Data Box Blob storage via http or https
  • Copy data to Data Box
  • Prepare to ship

Prerequisites

Before you begin, make sure that:

  1. You've completed the Tutorial: Set up Azure Data Box.
  2. You've received your Data Box and the order status in the portal is Delivered.
  3. You've reviewed the system requirements for Data Box Blob storage and are familiar with supported versions of APIs, SDKs, and tools.
  4. You've access to a host computer that has the data that you want to copy over to Data Box. Your host computer must
    • Run a Supported operating system.
    • Be connected to a high-speed network. We strongly recommend that you have at least one 10-GbE connection. If a 10-GbE connection isn't available, a 1-GbE data link can be used but the copy speeds will be impacted.
  5. Download AzCopy 7.1.0 on your host computer. You'll use AzCopy to copy data to Azure Data Box Blob storage from your host computer.

Connect to Data Box Blob storage

You can connect to Data Box Blob storage over http or https. In general, https is the secure and recommended way to connect to Data Box Blob storage. Http is used when connecting over trusted networks. Depending on whether you are connecting to Data Box Blob storage over http or https, the steps can be different.

Connect via http

Connection to Data Box Blob storage REST APIs over http requires the following steps:

  • Add the device IP and blob service endpoint to the remote host
  • Configure third-party software and verify the connection

Each of these steps is described in the following sections.

Add device IP address and blob service endpoint to the remote host

  1. Sign into the Data Box device. Ensure it is unlocked.

    Data Box dashboard

  2. Go to Set network interfaces. Make a note of the device IP address for the network interface used to connect to the client.

    Data Box dashboard

  3. Go to Connect and copy and click Rest (Preview).

    Data Box dashboard

  4. From the Access Storage account and upload data dialog, copy the Blob Service Endpoint.

    Data Box dashboard

  5. Start Notepad as an administrator, and then open the hosts file located at C:\Windows\System32\Drivers\etc.

  6. Add the following entry to your hosts file: <device IP address> <Blob service endpoint>
  7. For reference, use the following image. Save the hosts file.

    Data Box dashboard

Configure partner software and verify connection

Configure the partner software to connect to the client. To connect to the partner software, you would typically need the following information (may vary) you gathered from the Connect and copy page of the local web UI in the previous step:

  • Storage account name
  • Access key
  • Blob service endpoint

To verify that the connection is successfully established, use Storage Explorer to attach to an external storage account. If you do not have Storage Explorer, download Storage Explorer.

If this is the first time you are using Storage Explorer, you need to perform the following steps.

  1. From the top command bar, go to Edit > Target Azure Stack.

    Data Box dashboard

  2. Restart the Storage Explorer for the changes to take effect.

    Data Box dashboard

Follow these steps to connect to the storage account and verify the connection.

  1. In Storage Explorer, open the Connect to Azure Storage dialog. In the Connect to Azure Storage dialog, select Use a storage account name and key.

    Data Box dashboard

  2. Paste your Account name and Account key (key 1 value from the Connect and copy page in the local web UI). Select Storage endpoints domain as Other (enter below) and then provide the blob service endpoint as shown below. Check Use HTTP option only if transferring over http. If using https, leave the option unchecked. Select Next.

    Data Box dashboard

  3. In the Connection Summary dialog, review the provided information. Select Connect.

    Data Box dashboard

  4. The account that you successfully added is displayed in the left pane of Storage Explorer with (External, Other) appended to its name. Click Blob Containers to view the container.

    Data Box dashboard

Connect via https

Connection to Azure Blob storage REST APIs over https requires the following steps:

  • Download the certificate from Azure portal
  • Prepare the host computer for remote management
  • Add the device IP and blob service endpoint to the remote host
  • Configure third-party software and verify the connection

Each of these steps is described in the following sections.

Download certificate

Use the Azure portal to download certificate.

  1. Sign into the Azure portal.
  2. Go to your Data Box order and navigate to General > Device details.
  3. Under Device credentials, go to API access to device. Click Download. This action downloads a .cer certificate file. Save this file. You will install this certificate on the client or host computer that you will use to connect to the device.

    Download certificate in Azure portal

Prepare the host for remote management

Follow these steps to prepare the Windows client for a remote connection that uses an https session:

  • Import the .cer file into the root store of the client or remote host.
  • Add the device IP address and blob service endpoint to the hosts file on your Windows client.

Each of the preceding procedures, is described below.

Import the certificate on the remote host

You can use Windows PowerShell or the Windows Server UI to import and install the certificate on your host system.

Using PowerShell

  1. Start a Windows PowerShell session as an administrator.
  2. At the command prompt, type:

    Import-Certificate -FilePath C:\temp\localuihttps.cer -CertStoreLocation Cert:\LocalMachine\Root
    

Using Windows Server UI

  1. Right-click the .cer file and select Install certificate. This starts the Certificate Import Wizard.
  2. For Store location, select Local Machine, and then click Next.

    Import certificate using PowerShell

  3. Select Place all certificates in the following store, and then click Browse. Navigate to the root store of your remote host, and then click Next.

    Import certificate using PowerShell

  4. Click Finish. A message that tells you that the import was successful appears.

    Import certificate using PowerShell

To add device IP address and blob service endpoint to the remote host

The steps to follow are identical to what you used while connecting over http.

Configure partner software to establish connection

The steps to follow are identical to what you used while connecting over http. The only difference is that you should leave the Use http option unchecked.

Copy data to Data Box

Once you are connected to the Data Box Blob storage, the next step is to copy data. Prior to data copy, review the following considerations:

  • While copying data, ensure that the data size conforms to the size limits described in the Azure storage and Data Box limits.
  • If data, which is being uploaded by Data Box, is concurrently uploaded by other applications outside of Data Box, this may result in upload job failures and data corruption.

In this tutorial, AzCopy is used to copy data to Data Box Blob storage. You can also use Azure Storage Explorer (if you prefer a GUI-based tool) or a partner software to copy the data. The copy procedure has the following steps:

  • Create a container
  • Upload contents of a folder to Data Box Blob storage
  • Upload modified files to Data Box Blob storage

Each of these steps is described in detail in the following sections.

Create a container

The first step is to create a container, because blobs are always uploaded into a container. Containers organize groups of blobs like you organize files in folders on your computer. Follow these steps to create a blob container.

  1. Open Storage Explorer.
  2. In the left pane, expand the storage account within which you wish to create the blob container.
  3. Right-click Blob Containers, and from the context menu, select Create Blob Container.

    Create blob containers context menu

  4. A text box appears below the Blob Containers folder. Enter the name for your blob container. See the Create the container and set permissions for information on rules and restrictions on naming blob containers.

  5. Press Enter when done to create the blob container, or Esc to cancel. Once the blob container is successfully created, it is displayed under the Blob Containers folder for the selected storage account.

    Blob container created

Upload contents of a folder to Data Box Blob storage

Use AzCopy to upload all files in a folder to Blob storage on Windows or Linux. To upload all blobs in a folder, enter the following AzCopy command:

Linux

azcopy \
    --source /mnt/myfolder \
    --destination https://data-box-storage-account-name.blob.device-serial-no.microsoftdatabox.com/container-name/files/ \
    --dest-key <key> \
    --recursive

Windows

AzCopy /Source:C:\myfolder /Dest:https://data-box-storage-account-name.blob.device-serial-no.microsoftdatabox.com/container-name/files/ /DestKey:<key> /S

Replace <key> with your account key. To get your account key, in the Azure portal, go to your storage account. Go to Settings > Access keys, select a key, and paste it into the AzCopy command.

If the specified destination container does not exist, AzCopy creates it and uploads the file into it. Update the source path to your data directory, and replace data-box-storage-account-name in the destination URL with the name of the storage account associated with your Data Box.

To upload the contents of the specified directory to Blob storage recursively, specify the --recursive (Linux) or /S (Windows) option. When you run AzCopy with one of these options, all subfolders and their files are uploaded as well.

Upload modified files to Data Box Blob storage

Use AzCopy to upload files based on their last-modified time. To try this, modify or create new files in your source directory for test purposes. To upload only updated or new files, add the --exclude-older (Linux) or /XO (Windows) parameter to the AzCopy command.

If you only want to copy source resources that do not exist in the destination, specify both --exclude-older and --exclude-newer (Linux) or /XO and /XN (Windows) parameters in the AzCopy command. AzCopy uploads only the updated data, based on its time stamp.

Linux

azcopy \
--source /mnt/myfolder \
--destination https://data-box-storage-account-name.blob.device-serial-no.microsoftdatabox.com/container-name/files/ \
--dest-key <key> \
--recursive \
--exclude-older

Windows

AzCopy /Source:C:\myfolder /Dest:https://data-box-storage-account-name.blob.device-serial-no.microsoftdatabox.com/container-name/files/ /DestKey:<key> /S /XO

Prepare to ship

Final step is to prepare the device to ship. In this step, all the device shares are taken offline. The shares cannot be accessed once you start preparing the device to ship.

  1. Go to Prepare to ship and click Start preparation.

    Prepare to ship 1

  2. By default, checksums are computed inline during the prepare to ship. The checksum computation may take some time depending upon the size of your data. Click Start preparation.

    1. The device shares go offline and the device is locked when we prepare to ship.

      Prepare to ship 1

    2. The device status updates to Ready to ship once the device preparation is complete.

      Prepare to ship 1

    3. Download the list of files (manifest) that were copied in this process. You can later use this list to verify the files uploaded to Azure.

      Prepare to ship 1

  3. Shut down the device. Go to Shut down or restart page and click Shut down. When prompted for confirmation, click OK to continue.

  4. Remove the cables. The next step is to ship the device to Microsoft.

Next steps

In this tutorial, you learned about Azure Data Box topics such as:

  • Connect to Data Box Blob storage via http or https
  • Copy data to Data Box
  • Prepare to ship

Advance to the next tutorial to learn how to ship your Data Box back to Microsoft.