question

SumanDutta-8904 avatar image
0 Votes"
SumanDutta-8904 asked PratikSomaiya commented

How 'Treat empty column as null feature' can be implemented in ADLS gen2 dataset in Azure Data Factory?

There is a feature as 'Treat empty column as null' in ADLS gen1 dataset. We are migrating ADLS gen1 to gen2 . In the ADLS gen2 dataset this property is not there. How can we implement this property in ADLS gen2 dataset? Also, what does this property does? I have attached as snapshot of this property in aDLS gen1 dataset..
197140-image.png


azure-data-factoryazure-data-lake-storage
image.png (14.8 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PratikSomaiya avatar image
0 Votes"
PratikSomaiya answered PratikSomaiya commented

Hello @SumanDutta-8904

You can use Azure Data Flow to apply the transformation as mentioned in this link


ADLS Gen 2 Dataset only have the following Dataset properties:

197297-image.png



image.png (26.6 KiB)
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

we cannot add data flow..it is a migration project ..Any idea what this property does?

0 Votes 0 ·

I can see that you are using a Delimited Text dataset in ADLS Gen 1, is this your migrated dataset? As I went through the documentation of ADLS Gen 1 Dataset Properties but can't see this property there, maybe it is removed in the latest version. Can you try creating a new dataset using ADLS Gen 1 and see if you are getting the same property? As both ADLS Gen 1 and Gen 2 are not supporting this property as per their documentation

@AnnuKumari-MSFT : any thoughts here?

0 Votes 0 ·

in latest version of ADLS gen1 dataset it is not there... From the Json I picked it up that dataset type
"type": "AzureDataLakeStoreFile"

where as in dataset it is like "type": "avro"/"delimtext"...
I think this ADLS gen1 dataset was created in ADF V1..We are trying to migrate it to ADLS gen2..

0 Votes 0 ·
Show more comments