question

PranayKumar-4740 asked Feb 17, '22 SadiqhAhmed-MSFT edited Feb 21, '22

How to handle seeding data to Data Drives and then implementing delta sync after the seeded data has been placed into Azure?

Hello,

How to handle seeding data to Data Drives and then implementing delta sync after the seeded data has been placed into Azure?

What we want to do -

 Move 100 TB of data in which 80TB will be placed in File Share; the other 20TB will be from SQL Server.

 We are planning to use 2 data boxes to seed the 100 TB

 After seeded in Azure, delta sync needs to be implemented for the changes to the data after the backup was performed on the data disks.

 We currently have 100MB connection to the internet to support the delta syncs.

Questions
1. Can the data boxes be used in this scenario? If not what are alternatives to support the above?
2. How does the data need to be configured in Azure to support delta sync from On-prem?
3. What tools can be used to manage the sync between On-prem SQL / File Share to the Data Boxes?

If data boxes are not supported then what are alternative options to meet the above requirements?

azure-sql-database azure-data-share azure-data-box-family

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

pituach answered Feb 17, '22 pituach commented Feb 17, '22

Hi,

Can the data boxes be used in this scenario? If not what are alternatives to support the above?

As much as I understand your description the answer is: partially

data boxes are physical devices, which you use to move data from and to the Azure by sniping the device.

Basically it is like using disk-on-key or external disk to move files from one machine to another by upload the data to hard disk in one location and and ship it (not using network but physically ship the disk) to the second location.

This is great solution for handle seeding data but probably less fit for keeping your data sync between two places.

If you plan to sync the data once every X days and you are OK with the latency of shipping the devices, then it can fit as well, but this not a common scenario probably.

If you need to keep the data synchronized more frequently and the amount of delta sync data is not large, then using this solution does not make sense, and you can use the network to pass the data. If you need to keep the data synchronized at all time, then this is solution do not do the work at all (your description is not clear regarding the synchronization you need).

How does the data need to be configured in Azure to support delta sync from On-prem?

The following document include the full explanation about the procedure

https://docs.microsoft.com/en-us/azure/databox/data-box-deploy-ordered?WT.mc_id=DP-MVP-5001699

What tools can be used to manage the sync between On-prem SQL / File Share to the Data Boxes?

Speaking about File Share do you mean Azure files?

Your scenario is not fully clear to me regarding SQL Server!

data box has nothing to do with sync databases but sync data on disk. I do not understand what is the direct relation to SQL Server. Obviously you can use shared folder (like Azure files) for example to configure transactional replication between Azure SQL Managed Instance and SQL Server, but in this case the files are pushed by the server using the network.

You can manually move backup files from local to shared folder in the Azure in order to manually sync database. IN theory in this case you might want to move the files using data box if this fit the golden roles I mentioned above in question 1

If you need to sync files between on-premises disk and Azure files then you can use the Azure File Sync agent which enables data on a Windows Server to be synchronized with an Azure File share.

https://docs.microsoft.com/en-us/azure/storage/file-sync/file-sync-deployment-guide?WT.mc_id=DP-MVP-5001699

Again, without more information I cannot know if this is what you need but this is the common scenario of keeping local folder sync with Azure files.

If data boxes are not supported then what are alternative options to meet the above requirements?

As I mentioned data box fit for part of the scenario like your first moving of the files but for sync files Azure File Sync agent should fit better in most cases and for sync database a replication is probably the best solution

Ronen Ariely
Personal Site | Blog | Facebook | Linkedin

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

PranayKumar-4740 · Feb 17 at 11:58 AM

Hello, thanks for the reply.

Basically for the SQL data the size is around 20 TB which grows everyday. So, the idea was to use a Azure data box to first migrate the bulk say 20 TB (which might take 1 week to get into azure considering the Data transfer and the shipping). and then sync the delta's. Is there a way of achieving this ?

For File share - 80 TB is the size of the file share where we wanted to transfer the bulk using the data box and then sync the delta's.

0 ·

pituach PranayKumar-4740 · Feb 17 at 04:42 PM

Hi,

use a Azure data box to first migrate the bulk say 20 TB

This make sense. As I said, for the first seed data the Azure Data Box can fits great.

and then sync the delta's. Is there a way of achieving this ?

For the sync it depend on the parameter I mentioned like what is the size of the delta data to sync, how to you sync it (do you mean to transfer the backup of the transaction log or Differential Backups? I assume that you do not mean to use full back each time to make the sync. but most important is the parameter of latency. Shipping the data disk take time and cost more than other solution.

For the sync, this is not a common solution and without more information, this will not be my first (or second or third) solution.

For File share - 80 TB is the size of the file share where we wanted to transfer the bulk using the data box and then sync the delta's.

If you need to transfer 80 TB then probably Data box is the best solution

Is there a way of achieving this ?

I am not sure what can be added to what I wrote already. There are multiple ways to sync SQL Server and transferring files is not the most the common solution, but it depend on your need and the parameters I mentioned (which you did not provided)

There is no "right way" for all cases. the "right way" depend on your specific scenario.

It seems like data box can fit your need for the first transfering of the data, but for the sync we need more information in order to think if it fit as I mentioned

0 ·