Data flow sink preview is sorted but not sorted in the outputted tsv sink file

rbc_user 0 Reputation points
2024-03-21T04:05:25.86+00:00

I have a data flow in data factory that gets data from a database and transforms some columns and then outputs it on a tsv sink file.

In this data flow the data is sorted by a certain column. The sink preview is sorted correctly but when I open the outputted tsv file the data is not sorted. Any suggestions on how I can fix this?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,549 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2024-03-21T09:08:43.3266667+00:00

    Hi rbc_user,

    Thank you for posting query in Microsoft Q&A Platform.

    You need to consider repartitioning, consider setting a single partition in the optimize tab on the transformation. Consider keeping the sort transformation as close to the sink as possible.

    Hope this helps. Please let me know if any further queries.

    Please consider hitting Accept Answer button. Accepted answers help community as well.