question

SunilGupta-9709 avatar image
0 Votes"
SunilGupta-9709 asked YutongTie-MSFT edited

Azure Data Factory datasets

Hi Team,


We are migrating from Azure Data lake Gen1 to Gen2 (Storage Account).

Currently Azure Data Factory contains 300+ datasets which are responsible to read/write to Data Lake Gen1. Now this has to change to Gen2 compatible.
Gen1 and Gen2 data sets are not interchangeable and their JSON also not support.

Could you please guide us any tools/technique which can change all Gen1 compatible datasets to Gen2 compatible Datasets?

It would be really very helpful.

Thanks
Sunil

azure-data-factoryazure-data-lake-storage
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

nasreen-akter avatar image
0 Votes"
nasreen-akter answered ShaikMaheer-MSFT commented

Hi @SunilGupta-9709,

Thank you for the ask. I think you could do something like:

  • create a new branch e.g., feature/changedLS

  • if you have a LinkedService named "LS_ADLS_Name" for DataLakeGen1, create a new LinkedService for the DataLakeGen2 with a different name e.g., "LS_ADLS_Name2"

  • now go to the source control --> feature/changedLS branch, delete the DS_ADLS_Name.json file and renamed the LS_ADLS_Name2.json to LS_ADLS_Name.json and also changed the name inside the file

That should do the trick. Thanks!


· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.


Thanks nasreen-akter for valuable input.

unfortunately this trick is not working.

We have lot of datasets (200+) and their property is not compatible now with Gen2 Datasets (looking the JSON structure).

It really become cumbersome to create all new Gen2 datasets & pipeline changes. We also have some dynamic datasets for which parameters need to update in pipeline. These things are still not support.

Microsoft have given deadlines for Gen2 upgradte (FEb 2024) but did'nt provide any tool which can update exisitng datasets and pipelines. Its a enormous efforts and customer will be reluctant for these efforts.

We also have AAS models which are current used Gen1 data source. We thought to update the source from Gen1 to Gen2 will work. However that is also not working. Import again tables and creation of measures is certainly a effort intensive tasks.

kindly provide any direction or reference .

Thanks
Sunil

0 Votes 0 ·

Hi @SunilGupta-9709 ,

Thanks for posting query in Microsoft Q&A Platform.

There is no direct GUI way of converting Gen1 datasets to Gen2 datasets in ADF.

We need to consider re-creating them manually or you can also try to implement some automation using Azure data factory SDKs.

Idea of playing with json files in repository also looks fine to me. But we need to be more careful that json properties as properly pointing to Gen2 type.

0 Votes 0 ·