Partitioning inline delta lake sink

Question

question

SimoneGraziani-9114 asked Oct 21, '21 SimoneGraziani-9114 commented Oct 26, '21

Partitioning inline delta lake sink

Hello,

I'm trying to set the partitioning for a delta lake sink in Data Factory; as you can see in the attached screenshot there is only one option (Key partitioning) and I cannot select the field to use as key. Ideally, I would like to use a hash partitioning scheme with 10 partitions based on a specific field. We also tried to edit directly the dataflow script to specify the partitioning options but it doesn't seem to work.

Is the partitioning even supported by inline delta lake? If so, how can I use it? Are we missing something?

Thank you.

Best,
Simone

azure-data-factory

partition-option.png (28.6 KiB)

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2021-10-22T23:03:35.28Z

MarkKromer-MSFT answered Oct 22, '21 SimoneGraziani-9114 commented Oct 26, '21

There was a bug in the browser UI that was recently patched. For now, you can change the type of sink to something else, like a delimited text sink, and you should then see the key columns in Key partitioning. Then, switch the Sink type back to Delta.

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

SimoneGraziani-9114 · Oct 26, 2021 at 07:59 AM

Thank you @MarkKromer-MSFT ! Indeed it works, but the only partitioning option is Key partitioning. Is the fixed partitioning available? I know that I could simulate it with Key partitioning and a calculated field, but that wouldn't be optimal when using Power BI to query the data (I would have to add a filter on the calculated field).

0 ·

question