Memory optimized compute type for Data Flows in Azure Data Factory and Azure Synapse
APPLIES TO:
Azure Data Factory
Azure Synapse Analytics
Data flow activities in Azure Data Factory and Azure Synapse support the Compute type setting to help optimize the cluster configuration for cost and performance of the workload. The default selection for the setting is General and will be sufficient for most data flow workloads. General purpose clusters typically provide the best balance of performance and cost. However, the Memory optimized setting can significantly improve performance in some scenarios by maximizing the memory available per core for the cluster.
When to use the memory optimized compute type
If your data flow has many joins and lookups, you may want to use a memory optimized cluster. These more memory intensive operations will benefit particularly by additional memory, and any out-of-memory errors encountered with the default compute type will be minimized. Memory optimized clusters do incur the highest cost per core, but may avoid pipeline failures for memory intensive operations. If you experience any out of memory errors when executing data flows, switch to a memory optimized Azure IR configuration.