Migrate data to Azure with Data Dynamics StorageX

Article
10/03/2021

This article helps you deploy Data Dynamics StorageX in Microsoft Azure. We introduce key concepts around how StorageX works, deployment prerequisites, installation process, and how-tos for operational guidance. For more in-depth information, visit Data Dynamics Customer Portal.

Data Dynamics StorageX is a Unified Unstructured Data Management platform that allows analyzing, managing, and moving data across heterogenous storage environments. Basic capabilities are:

Data Movement capabilities
- NAS to NAS (with or without Analysis)
- NAS to Object replication
- NAS to Object archive
Duplicate file check report
Open shares report
File Meta Data Analysis with custom tagging

You can find more functionalities on StorageX, and comparison with similar tools in our comparison table.

This document covers three major topics:

Process for deploying the StorageX offering in Azure environment,
Basic steps for adding file and cloud storage resources to StorageX,
Creating Data Movement Policies.

Reference architecture

The following diagram provides a reference architecture for on-premises to Azure and in-Azure deployments of StorageX.

Diagram that shows reference architecture for StorageX implementation

StorageX interoperability matrix

Feature	SMB	NFS	Notes
Migration	SMB to SMB	NFSv3 to NFSv3	Can be combined for multi-protocol
File-to-object archive	SMB to Azure Blob Storage	NFSv3 to Azure Blob Storage	Archive data to optimize cost
File-to-object replication	SMB to Azure Blob Storage	NFSv3 to Azure Blob Storage	Replicate data to increase protection

Before you begin

Upfront planning will ease the migration and reduce the risks. Some of tips to ease the migration are:

Gather data on the source and targets services,
Make sure you have a list of your connectivity points and all necessary network ports.
Your StorageX server in Azure will need to be part of your Active Directory infrastructure. Ensure you engage an administrator with Active Directory privileges to add the server to Active Directory.
For easier implementation use Azure Marketplace.

Get started with Azure

Microsoft offers a framework to follow to get you started with Azure. The Cloud Adoption Framework (CAF) is a detailed approach to enterprise digital transformation and comprehensive guide to planning a production grade cloud adoption. The CAF includes a step-by-step Azure setup guide to help you get up and running quickly and securely. You can find an interactive version in the Azure portal. You'll find sample architectures, specific best practices for deploying applications, and free training resources to put you on the path to Azure expertise.

Considerations for migrations

Several aspects are important when considering migrations of file data to Azure. Before proceeding learn more:

Storage migration overview
latest supported features by Data Dynamics StorageX in migration tools comparison matrix.

Remember, you'll require enough network capacity to support migrations without impacting production applications. This section outlines the tools and techniques that are available to assess your network needs.

Determine unutilized internet bandwidth

It's important to know how much typically unutilized bandwidth (or headroom) you have available on a day-to-day basis. To help you assess whether you can meet your goals for:

initial time for migrations when you're not using Azure Data Box for offline method
time required to do incremental resync before final switch-over to the target file service

Use the following methods to identify the bandwidth headroom to Azure that is free to consume.

If you're an existing Azure ExpressRoute customer, view your circuit usage in the Azure portal.
Contact your ISP and request reports to show your existing daily and monthly utilization.
There are several tools that can measure utilization by monitoring your network traffic at the router/switch level:

Creating a StorageX service account

For SMB migrations, service account to run the StorageX services is recommended. Service account enables access to be traced for support reasons, while giving the proper privileges to grant full access.

The process that runs data movement policies on the StorageX universal data engines uses the StorageX service account. Account must have all necessary privileges assigned to it in the Active Directory domain.

You can create an account like domain administrator if allowed in your environment. Recommended approach is to grant permissions directly on the NAS server. To do the latter, place the service account in the Administrators and Backup operators groups.

The StorageX service account requires the following privileges:

SeServiceLogonRight
SeSecurityPrivilege
SeBackupPrivilege
SeRestorePrivilege
SeAuditPrivilege
SeInteractiveLogonRight

NFS access requires an export rule for the IP of the StorageX server, and grant access for RW and root. For example, "root=10.2.3.12, rw=10.2.3.12", where 10.2.3.12 is the StorageX server IP address.

Deployment guide

This section provides guidance on how to deploy a StorageX server, and other StorageX components in your Azure environment.

Before implementation starts, several prerequisites are important:

Target storage service must be identified and created,
Make sure you have the proper ports open to access the storage services. You can use Microsoft’s portqryui tool to verify necessary ports are open.

You can learn more here:

How to create Azure File Share
How to create an SMB volume or NFS export in Azure NetApp Files

Any SMB migrations require Active Directory to be properly set up before adding any resources. We will be using an existing Azure NetApp Files implementation with new volume for migration target. Before we can create a new Azure NetApp Files volume, we need to:

Once the configuration for Azure NetApp Files is done, we can create a share we will use for migration.

Navigate to existing Azure NetApp Files account, and select Volumes. Select Add volume. Give it a name, select capacity pool and networking setup.
Select SMB protocol, select existing Active Directory connection, and provide a Share name.
Add Tags if you use them, and Create a share.

Once the volume has been created, we can proceed with StorageX deployment.
Deploy the Data Dynamics Data Movement and Mobility offering in Azure. Recommended approach is using Azure Marketplace. Add the deployed server into your Active Directory domain.
On the deployed server, verify that all required network ports are open.
Run the StorageX installation script. Installation script is available on the desktop when you log in to your deployed server in Azure. All installation files and output logs are located in the folder C:\ProgramData\data Dynamics\StorageX. Within that folder, the InstallationFiles subfolder contains the zipped source installation files, while the SilentInstaller-DD folder contains the configuration XML files and installation logs.

By default, the installation script deploys the following services:
- StorageX
- StorageX API
- StorageX Processing Engine
- StorageX Universal Data Engine
- StorageX File Recovery
- Elasticsearch
- Microsoft SQL Express
- Microsoft SQL Management Studio
- Kibana
- Any supporting software (for example, Java)
The installation script has an XML configuration file where these different services can be customized, but we do not recommend manually modifying the configuration.

To install, right-click the StorageXPOCSilentInstaller - Shortcut on the Desktop and select Run as administrator. After the installation starts, you will be asked for the StorageX service account and the password for that account.

Installation script will continue installing all the services. When the installation process is finished, you will see the message on the command-line interface.

StorageX will start automatically.

Migration guide

Once the installation finishes, and all services are started, we can start migration process.

Open the StorageX Console.
Click the Storage Resources tab.
In the My Resources view, add your NAS devices or storage services, called storage resources in StorageX. You can add different types:
- Add file storage resource for on-premises NAS devices, Windows file server, Azure NetApp Files, or generic file servers,
- Add object storage resource for S3 or Azure Blob Storage (including Azure Files service),
- Add DFS namespace for DFS file servers.
Select Add file storage resource to start the wizard of adding Azure NetApp Files migration target.
Select Generic CIFS for CIFS and SMB migrations.
Browse to find Azure NetApp Files computer account. Azure NetApp Files computer account was created when it was added to Active Directory Domain.

Finish the wizard, and verify the account has been added, and contains the share we created.

Every source, and target must be added as a separate resource. Example shows adding Azure NetApp Files share, and it can be slightly different for other services.
Click the Data Movement tab.
In the tree pane, right-click the folder where you want to create the new policy and select Phased Migration Policy.
Follow the steps in the wizard to create a Phased Migration policy. Select appropriate Source and Destination targets.
Provide a name and description for the policy and Finish the wizard.
Right-click the new policy and select Run to run a test migration. We recommend checking the security configuration on multiple files on the source resource to verify necessary permissions have been provided.
Check the status of the policy to make sure initial migration has run successfully.

Support

If issues occur, Microsoft and Data Dynamics can provide help using regular support channels. Contact Microsoft for infrastructure issues or with any Azure service. Contact Data Dynamics for issues tied to Data Dynamics StorageX installation.

How to open a case with Azure

In the Azure portal search for support in the search bar at the top. Select Help + support -> New Support Request.

How to open a case with Data Dynamics

Go to the Data Dynamics Support Portal. If you have not registered, provide your email address, and our Support team will create an account for you. Once you have signed in, open a user request. If you have already opened an Azure support case, note support request number when creating the request.

Next steps

Various resources are available to learn more:

Storage migration overview
Features supported by Data Dynamics StorageX in migration tools comparison matrix
Data Dynamics
Data Dynamics Customer Portal contains full documentation for StorageX