Azure Data Factory datasets

Question

question

SunilGupta-9709 asked Apr 13, '22 YutongTie-MSFT edited May 2, '22

Azure Data Factory datasets

Hi Team,

We are migrating from Azure Data lake Gen1 to Gen2 (Storage Account).

Currently Azure Data Factory contains 300+ datasets which are responsible to read/write to Data Lake Gen1. Now this has to change to Gen2 compatible.
Gen1 and Gen2 data sets are not interchangeable and their JSON also not support.

Could you please guide us any tools/technique which can change all Gen1 compatible datasets to Gen2 compatible Datasets?

It would be really very helpful.

Thanks
Sunil

azure-data-factory azure-data-lake-storage

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2022-04-13T15:11:54.23Z

nasreen-akter answered Apr 13, '22 ShaikMaheer-MSFT commented Apr 20, '22

Hi @SunilGupta-9709,

Thank you for the ask. I think you could do something like:

create a new branch e.g., feature/changedLS
if you have a LinkedService named "LS_ADLS_Name" for DataLakeGen1, create a new LinkedService for the DataLakeGen2 with a different name e.g., "LS_ADLS_Name2"
now go to the source control --> feature/changedLS branch, delete the DS_ADLS_Name.json file and renamed the LS_ADLS_Name2.json to LS_ADLS_Name.json and also changed the name inside the file

That should do the trick. Thanks!

· 2

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SunilGupta-9709 · Apr 15 at 11:50 AM

Thanks nasreen-akter for valuable input.

unfortunately this trick is not working.

We have lot of datasets (200+) and their property is not compatible now with Gen2 Datasets (looking the JSON structure).

It really become cumbersome to create all new Gen2 datasets & pipeline changes. We also have some dynamic datasets for which parameters need to update in pipeline. These things are still not support.

Microsoft have given deadlines for Gen2 upgradte (FEb 2024) but did'nt provide any tool which can update exisitng datasets and pipelines. Its a enormous efforts and customer will be reluctant for these efforts.

We also have AAS models which are current used Gen1 data source. We thought to update the source from Gen1 to Gen2 will work. However that is also not working. Import again tables and creation of measures is certainly a effort intensive tasks.

kindly provide any direction or reference .

Thanks
Sunil

0 ·

ShaikMaheer-MSFT SunilGupta-9709 · Apr 20 at 03:28 PM

Hi @SunilGupta-9709 ,

Thanks for posting query in Microsoft Q&A Platform.

There is no direct GUI way of converting Gen1 datasets to Gen2 datasets in ADF.

We need to consider re-creating them manually or you can also try to implement some automation using Azure data factory SDKs.

Idea of playing with json files in repository also looks fine to me. But we need to be more careful that json properties as properly pointing to Gen2 type.

0 ·

question