Hi Team,
We are looking at approach to copy data on a daily basis from one ADLS Gen2 storage account to another ADLS Gen2 storage account. Containers in our storage account are dynamically growing and folder structure is not pre-defined. We need to append timestamp to the file name during the copy operation as we need to retain only 30 copies of every file (last 30days).
We consider below approaches but ran into challenges
AzCopy - Recursive file copy allowed but rename operation on files is not permitted
ADF - Limitation on recursive file copy as we need to decide on the sub folders layers initially to trigger copy data activity of files inside foreach loops for containers
Snapshots - It can create the file only in the same storage account but we need to have copy of the file in different storage account
Please suggest the suitable workaround for the same.
Thanks,
Haritha