question

SimoneGraziani-9114 avatar image
0 Votes"
SimoneGraziani-9114 asked SimoneGraziani-9114 commented

Partitioning inline delta lake sink

Hello,

I'm trying to set the partitioning for a delta lake sink in Data Factory; as you can see in the attached screenshot there is only one option (Key partitioning) and I cannot select the field to use as key. Ideally, I would like to use a hash partitioning scheme with 10 partitions based on a specific field. We also tried to edit directly the dataflow script to specify the partitioning options but it doesn't seem to work.

Is the partitioning even supported by inline delta lake? If so, how can I use it? Are we missing something?

Thank you.

Best,
Simone

142511-partition-option.png


azure-data-factory
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

MarkKromer-MSFT avatar image
0 Votes"
MarkKromer-MSFT answered SimoneGraziani-9114 commented

There was a bug in the browser UI that was recently patched. For now, you can change the type of sink to something else, like a delimited text sink, and you should then see the key columns in Key partitioning. Then, switch the Sink type back to Delta.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you @MarkKromer-MSFT ! Indeed it works, but the only partitioning option is Key partitioning. Is the fixed partitioning available? I know that I could simulate it with Key partitioning and a calculated field, but that wouldn't be optimal when using Power BI to query the data (I would have to add a filter on the calculated field).

0 Votes 0 ·