Analyze and migrate to Azure with Komprise
This article helps you integrate the Komprise Intelligent Data Management infrastructure with Azure storage services. It includes considerations and implementation guidance on how to analyze, and migrate your data.
Komprise provides analytics and insights into file, and object data stored in network attached storage systems (NAS), and object stores, both on-premises and in the cloud. It enables migration of data to Azure storage services like Azure Files, Azure NetApp Files, Azure Blob Storage, or other ISV NAS solution. Learn more on verified partner solutions for primary and secondary storage.
Common use cases for Komprise include:
- Analysis of unstructured file and object data to gain insights for data management, movement, positioning, archiving, protection, and confinement,
- Migration of file data to Azure Files, Azure NetApp Files, or ISV NAS solution,
- Policy based tiering and archiving of file data to Azure Blob Storage while retaining transparent access from the original NAS solution and allowing native object access in Azure,
- Copy file data to Azure Blob Storage on configurable schedules while retaining native object access in Azure
- Migration of object data to Azure Blob Storage,
- Tiering and data lifecycle management of objects across Hot, Cool, and Archive tiers of Azure Blob Storage based on last access time
The following diagram provides a reference architecture for on-premises to Azure and in-Azure deployments.
The following diagram provides a reference architecture for migrating cloud and on-premises object workloads to Azure Blob Storage.
Komprise is a software solution that is easily deployed in a virtual environment. The solutions consist of:
- Director - The administration console for the Komprise Grid. It is used to configure the environment, monitor activities, view reports and graphs, and set policies.
- Observers - Manage and analyze shares, summarize reports, communicate with the Director, and handle object and NFS data traffic.
- Proxies - Simplify and accelerate SMB/CIFS data flow, easily scale to meet performance requirements of a growing environment.
Before you begin
Upfront planning will help in migrating the data with less risk.
Get started with Azure
Microsoft offers a framework to follow to get you started with Azure. The Cloud Adoption Framework (CAF) is a detailed approach to enterprise digital transformation and comprehensive guide to planning a production grade cloud adoption. The CAF includes a step-by-step Azure setup guide to help you get up and running quickly and securely. You can find an interactive version in the Azure portal. You'll find sample architectures, specific best practices for deploying applications, and free training resources to put you on the path to Azure expertise.
Considerations for migrations
Several aspects are important when considering migrations of file data to Azure. Before proceeding learn more:
- Storage migration overview
- latest supported features by Komprise Intelligent Data Management in migration tools comparison matrix.
Remember, you'll require enough network capacity to support migrations without impacting production applications. This section outlines the tools and techniques that are available to assess your network needs.
Determine unutilized internet bandwidth
It's important to know how much typically unutilized bandwidth (or headroom) you have available on a day-to-day basis. To help you assess whether you can meet your goals for:
- initial time for migrations when you're not using Azure Data Box for offline method
- time required to do incremental resync before final switch-over to the target file service
Use the following methods to identify the bandwidth headroom to Azure that is free to consume.
- If you're an existing Azure ExpressRoute customer, view your circuit usage in the Azure portal.
- Contact your ISP and request reports to show your existing daily and monthly utilization.
- There are several tools that can measure utilization by monitoring your network traffic at the router/switch level:
Migration planning guide
Komprise is simple to set up and enables running multiple migrations simultaneously in three steps:
- Analyze your data to identify files and objects to migrate or archive,
- Define policies to migrate, move, or copy unstructured data to Azure Storage,
- Activate policies that automatically move your data.
The first step is critical in finding and prioritizing the right data to migrate. Komprise analysis provides:
Information on access time to identify:
- Less frequently accessed files that you can cache on-premises or store on fast file service
- Cold data you can archive to blob storage
Information on top users, groups, or shares to determine the order of the migration and the most impacted group within the organization to assess business impact
Number of files, or capacity per file type to determine type of stored files and if there are any possibilities to clean up the content. Cleaning up will reduce the migration effort, and reduce the cost of the target storage. Similar analytics is available for object data.
Number of files, or capacity per file size to determine the duration of migration. Large number of small files will take longer to migrate than small number of large files. Similar analytics is available for object data.
Cost of objects by storage tier to determine if cold data is incorrectly placed in expensive tiers, or hot data is incorrectly placed in cheaper tiers with high access costs. Right placing data based on access patterns enables optimizing overall cloud storage costs.
Custom query capability filter to filter exact set of files and objects for your specific needs
Before deploying Komprise, the target service must be deployed. You can learn more here:
The Komprise Grid is deployed in a virtual environment (Hyper-V, VMware, KVM) for speed, scalability, and resilience. Alternatively, you may set up the environment in your Azure subscription using Azure Marketplace.
Open the Azure portal, and search for storage accounts.
You can also click on the default Storage accounts icon.
Select Create to add an account:
- Select existing resource group or Create new
- Provide a unique name for your storage account
- Choose the region
- Select Standard or Premium performance, depending on your needs. If you select Premium, select File shares under Premium account type.
- Choose the Redundancy that meets your data protection requirements
Next, we recommend the default settings from the Advanced screen. If you are migrating to Azure Files, we recommend enabling Large file shares if available.
Keep the default networking options for now and move on to Data protection. You can choose to enable soft delete, which allows you to recover an accidentally deleted data within the defined retention period. Soft delete offers protection against accidental or malicious deletion.
Add tags for organization if you use tagging and Create your account.
Two quick steps are all that are now required before you can add the account to your Komprise environment. Navigate to the account you created in the Azure portal and select File shares under the File service menu. Add a File share and choose a meaningful name. Then, navigate to the Access keys item under Settings and copy the Storage account name and one of the two access keys. If the keys are not showing, click on the Show keys.
Navigate to the Properties of the Azure File share. Write down the URL address, it will be required to add the Azure connection into the Komprise target file share:
(Optional) You can add extra layers of security to your deployment.
Configure role-based access to limit who can make changes to your storage account. For more information, see Built-in roles for management operations.
Restrict access to the account to specific network segments with storage firewall settings. Configure firewall settings to prevent access from outside of your corporate network.
Set a delete lock on the account to prevent accidental deletion of the storage account.
Configure extra security best practices.
Deployment instructions for managing file data
Download the Komprise Observer virtual appliance from the Director, deploy it to your hypervisor and configure it with the network and domain. Director is provided as a cloud service managed by Komprise. Information needed to access Director is sent with the welcome email once you purchase the solution.
To add the shares to analyze and migrate, you have two options:
Discover all the shares in your storage environment by entering:
- Platform for the source NAS
- Hostname or IP address
- Display name
- Credentials (for SMB shares)
Specify a file share by entering:
- Storage information
- Display Name
- Credentials (for SMB shares)
This step must be repeated to add other source and destination shares. To add Azure Files as a destination, you need to provide the Azure storage account and file share details:
Deployment instructions for managing object data
Managing object provides different experience. The Director and Observer are provided as a cloud services, managed by Komprise. If you only need to analyze and archive data in Azure Blob Storage, no further deployment is required. If you need to perform migrations into Azure Blob Storage, get the Komprise Observer virtual appliance sent with the welcome email, and deploy it in a Linux virtual machine in your Azure cloud infrastructure. After deploying, follow the steps on the Komprise Director.
Navigate to Data Stores and Add New Object Store. Select Microsoft Azure as the provider.
Add shares to analyze and migrate. These steps must be repeated for every source, and target share, or container. There are two options to perform the same action:
Discover all the containers by entering:
- Storage account name
- Primary access key
- Display name
Required information can be found in Azure Portal by navigating to the Access keys item under Settings for the storage account. If the keys are not showing, click on the Show keys.
Specify a container by entering:
- Container name
- Storage account name
- Primary access key
- Display name
Container name represents the target container for the migration and needs to be created before migration. Other required information can be found in Azure Portal by navigating to the Access keys item under Settings for the storage account. If the keys are not showing, click on the Show keys.
Komprise provides live migration, where end users and applications are not disrupted and can continue to access data during the migration. The migration process automates migrating directories, files, and links from a source to a destination. At each step data integrity is checked. All attributes, permissions, and access controls from the source are applied. In an object migration, objects, prefixes, and metadata of each object are migrated.
To configure and run a migration, follow these steps:
Log into your Komprise console. Information needed to access the console is sent with the welcome email once you purchase the solution.
Navigate to Migrate and click on Add Migration.
Add migration task by selecting proper source and destination share. Provide a migration name. Once configured, click on Start Migration. This step is slightly different for file and object data migrations.
File migration provides options to preserve access time and SMB ACLs on the destination. This option depends on the selected source and destination file service and protocol.
Object migration provides options to choose the destination Azure storage tier (Hot, Cool, Archive). You may also choose to verify each data transfer using MD5 checksum. Egress costs can occur with MD5 checksums as cloud objects must be retrieved to calculate the MD5 checksum.
Once the migration started, you can go to Migrate to monitor the progress.
Once all changes have been migrated, run one final migration by clicking on Actions and selecting Start final iteration. Before final migration, we recommend stopping access to source file shares or moving them to read-only mode (for users and applications). This step will make sure no changes happen on the source.
Once the final migration finishes, transition all users and applications to the destination share. Switch over to the new file service usually requires changing the configuration of DNS servers, DFS servers, or changing the mount points to the new destination.
As a last step, mark the migration completed.
To open a case with Komprise, sign in to the Komprise support site
Get Komprise listing on Azure Marketplace.
Various resources are available to learn more: