What's new in Azure Data Factory

Azure Data Factory is improved on an ongoing basis. To stay up to date with the most recent developments, this article provides you with information about:

  • The latest releases.
  • Known issues.
  • Bug fixes.
  • Deprecated functionality.
  • Plans for changes.

This page is updated monthly, so revisit it regularly.

May 2022


Service categoryService improvementsDetails
Data flowUser Defined Functions for mapping data flowsAzure Data Factory introduces in public preview user defined functions and data flow libraries. A user defined function is a customized expression you can define to be able to reuse logic across multiple mapping data flows. User defined functions live in a collection called a data flow library to be able to easily group up common sets of customized functions.
Learn more

April 2022


Service categoryService improvementsDetails
Data flowData preview and debug improvements in mapping data flowsDebug sessions using the AutoResolve Azure integration runtime (IR) will now start up in under 10 seconds. There are new updates to the data preview panel in mapping data flows. Now you can sort the rows inside the data preview view by selecting column headers. You can move columns around interactively. You can also save the data preview results as a CSV by using Export CSV.
Learn more
Dataverse connector is available for mapping data flowsDataverse connector is available as source and sink for mapping data flows.
Learn more
Support for user database schemas for staging with the Azure Synapse Analytics and PostgreSQL connectors in data flow sinkData flow sink now supports using a user database schema for staging in both the Azure Synapse Analytics and PostgreSQL connectors.
Learn more
MonitoringMultiple updates to Data Factory monitoring experiencesNew updates to the monitoring experience in Data Factory include the ability to export results to a CSV, clear all filters, and open a run in a new tab. Column and result caching is also improved.
Learn more
User interfaceNew regional format supportChoosing your language and the regional format in settings will influence the format of how data such as dates and times appear in the Azure Data Factory Studio monitoring. For example, the time format in Monitoring will appear like "Apr 2, 2022, 3:40:29 pm" when choosing English as the regional format, and "2 Apr 2022, 15:40:29" when choosing French as regional format. These settings affect only the Azure Data Factory Studio user interface and do not change/ modify your actual data and time zone.

March 2022


Service categoryService improvementsDetails
Data flowScriptLines and parameterized linked service support added mapping data flowsIt's now easy to detect changes to your data flow script in Git with ScriptLines in your data flow JSON definition. Parameterized linked services can now also be used inside your data flows for flexible generic connection patterns.
Learn more
Flowlets general availability (GA)Flowlets is now generally available to create reusable portions of data flow logic that you can share in other pipelines as inline transformations. Flowlets enable extract-transform-and-load (ETL) jobs to be composed of custom or common logic components.
Learn more
Change Feed connectors are available in five data flow source transformationsChange Feed connectors are available in data flow source transformations for Azure Cosmos DB, Azure Blob Storage, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, and the common data model (CDM).
Learn more
Data preview and debug improvements in mapping data flowsNew features were added to data preview and the debug experience in mapping data flows.
Learn more
SFTP connector for mapping data flowSFTP connector is available for mapping data flow as both source and sink.
Learn more
Data movementSupport Always Encrypted for SQL-related connectors in Lookup activity under Managed virtual networkAlways Encrypted is supported for SQL Server, Azure SQL Database, Azure SQL Managed Instance, and Synapse Analytics in the Lookup activity under managed virtual network.
Learn more
Integration runtimeNew UI layout in Azure IR creation and edit pageThe UI layout of the IR creation and edit page now uses tab style for Settings, Virtual network, and Data flow runtime.
Learn more
OrchestrationTransform data by using the Script activityYou can use a Script activity to invoke a SQL script in SQL Database, Azure Synapse Analytics, SQL Server, Oracle, or Snowflake.
Learn more
Web activity timeout improvementYou can configure response timeout in a Web activity to prevent it from timing out if the response period is more than one minute, especially in the case of synchronous APIs.
Learn more

February 2022


Service categoryService improvementsDetails
Data flowParameterized linked services supported in mapping data flowsYou can now use your parameterized linked services in mapping data flows to make your data flow pipelines generic and flexible.
Learn more
SQL Database incremental source extract available in data flow (public preview)A new option has been added on mapping data flow SQL Database sources called Enable incremental extract (preview). Now you can automatically pull only the rows that have changed on your SQL Database sources by using data flows.
Learn more
Four new connectors available for mapping data flows (public preview)Data Factory now supports four new connectors (public preview) for mapping data flows: Quickbase connector, Smartsheet connector, TeamDesk connector, and Zendesk connector.
Learn more
Azure Cosmos DB (SQL API) for mapping data flow now supports inline modeAzure Cosmos DB (SQL API) for mapping data flow can now use inline datasets.
Learn more
Data movementGet metadata-driven data ingestion pipelines on the Data Factory Copy Data tool within 10 minutes (GA)You can build large-scale data copy pipelines with a metadata-driven approach on the Copy Data tool within 10 minutes.
Learn more
Data Factory Google AdWords connector API upgrade availableThe Data Factory Google AdWords connector now supports the new AdWords API version. No action is required for the new connector user because it's enabled by default.
Learn more
Region expansionData Factory is now available in West US3 and Jio India WestData Factory is now available in two new regions: West US3 and Jio India West. You can colocate your ETL workflow in these new regions if you're using these regions to store and manage your modern data warehouse. You can also use these regions for business continuity and disaster recovery purposes if you need to fail over from another region within the geo.
Learn more
SecurityConnect to an Azure DevOps account in another Azure Active Directory (Azure AD) tenantYou can connect your Data Factory instance to an Azure DevOps account in a different Azure AD tenant for source control purposes.
Learn more

January 2022


Service categoryService improvementsDetails
Data flowQuick reuse is now automatic in all Azure IRs that use Time to Live (TTL)You no longer need to manually specify "quick reuse." Data Factory mapping data flows can now start up subsequent data flow activities in under five seconds after you set a TTL.
Learn more
Retrieve your custom Assert descriptionIn the Assert transformation, you can define your own dynamic description message. You can use the new function assertErrorMessage() to retrieve the row-by-row message and store it in your destination data.
Learn more
Automatic schema detection in Parse transformationA new feature added to the Parse transformation makes it easy to automatically detect the schema of an embedded complex field inside a string column. Select the Detect schema button to set your target schema automatically.
Learn more
Support Dynamics 365 connector as both sink and sourceYou can now connect directly to Dynamics 365 to transform your Dynamics data at scale by using the new mapping data flow connector for Dynamics 365.
Learn more
Always Encrypted SQL connections now available in data flowsAlways Encrypted can now source transformations in SQL Server, SQL Database, SQL Managed Instance, and Azure Synapse when you use data flows.
Learn more
Data movementData Factory Azure Databricks Delta Lake connector supports new authentication typesData Factory Databricks Delta Lake connector now supports two more authentication types: system-assigned managed identity authentication and user-assigned managed identity authentication.
Learn more
Data Factory Copy activity supports upsert in several more connectorsData Factory Copy activity now supports upsert while it sinks data to SQL Server, SQL Database, SQL Managed Instance, and Azure Synapse.
Learn more

December 2021


Service categoryService improvementsDetails
Data flowDynamics connector as native source and sink for mapping data flowsThe Dynamics connector is now supported as source and sink for mapping data flows.
Learn more
Native change data capture (CDC) is now natively supportedCDC is now natively supported in Data Factory for Azure Cosmos DB, Blob Storage, Data Lake Storage Gen1, Data Lake Storage Gen2, and CDM.
Learn more
Flowlets public previewThe flowlets public preview allows data flow developers to build reusable components to easily build composable data transformation logic.
Learn more
Map Data public previewThe Map Data preview enables business users to define column mapping and transformations to load Azure Synapse lake databases.
Learn more
Multiple output destinations from Power QueryYou can now map multiple output destinations from Power Query in Data Factory for flexible ETL patterns for citizen data integrators.
Learn more
External Call transformation supportExtend the functionality of mapping data flows by using the External Call transformation. You can now add your own custom code as a REST endpoint or call a curated third-party service row by row.
Learn more
Enable quick reuse by Azure Synapse mapping data flows with TTL supportAzure Synapse mapping data flows now support quick reuse by setting a TTL in the Azure IR. Using a setting enables your subsequent data flow activities to execute in under five seconds.
Learn more
Assert transformationEasily add data quality, data domain validation, and metadata checks to your Data Factory pipelines by using the Assert transformation in mapping data flows.
Learn more
IntelliSense support in expression builder for more productive pipeline authoring experiencesIntelliSense support in expression builder and dynamic content authoring makes Data Factory and Azure Synapse pipeline developers more productive while they write complex expressions in their data pipelines.
Learn more

November 2021


Service categoryService improvementsDetails
Continuous integration and continuous delivery (CI/CD) GitHub integration improvements Improvements in Data Factory and GitHub integration remove limits on 1,000 Data Factory resources per resource type, such as datasets and pipelines. For large data factories, this change helps mitigate the impact of the GitHub API rate limit.
Learn more
Data flowSet a custom error code and error message with the Fail activityFail activity enables ETL developers to set the error message and custom error code for a Data Factory pipeline.
Learn more
External call transformationMapping data flows External Call transformation enables ETL developers to use transformations and data enrichments provided by REST endpoints or third-party API services.
Learn more
Synapse quick reuseWhen you execute data flow in Synapse Analytics, use the TTL feature. The TTL feature uses the quick reuse feature so that sequential data flows will execute within a few seconds. You can set the TTL when you configure an Azure IR.
Learn more
Data movementCopy activity supports reading data from FTP or SFTP without chunkingAutomatically determine the file length or the relevant offset to be read when you copy data from an FTP or SFTP server. With this capability, Data Factory automatically connects to the FTP or SFTP server to determine the file length. After the length is determined, Data Factory divides the file into multiple chunks and reads them in parallel.
Learn more
UTF-8 without BOM support in Copy activityCopy activity supports writing data with encoding the type UTF-8 without BOM for JSON and delimited text datasets.
Multicharacter column delimiter supportCopy activity supports using multicharacter column delimiters for delimited text datasets.
Integration runtime Run any process anywhere in three steps with SQL Server Integration Services (SSIS) in Data Factory Learn how to use the best of Data Factory and SSIS capabilities in a pipeline. A sample SSIS package with parameterized properties helps you get a jump-start. With Data Factory Studio, the SSIS package can be easily dragged and dropped into a pipeline and used as part of an Execute SSIS Package activity.

This capability enables you to run the Data Factory pipeline with an SSIS package on self-hosted IRs or SSIS IRs. By providing run-time parameter values, you can use the powerful capabilities of Data Factory and SSIS capabilities together. This article illustrates three steps to run any process, which can be any executable, such as an application, program, utility, or batch file, anywhere.
Learn more

October 2021


Service categoryService improvementsDetails
Data flowAzure Data Explorer and Amazon Web Services (AWS) S3 connectorsThe Microsoft Data Integration team has released two new connectors for mapping data flows. If you're using Azure Synapse, you can now connect directly to your AWS S3 buckets for data transformations. In both Data Factory and Azure Synapse, you can now natively connect to your Azure Data Explorer clusters in mapping data flows.
Learn more
Power Query activity leaves preview for GAThe Data Factory Power Query pipeline activity is now generally available. This new feature provides scaled-out data prep and data wrangling for citizen integrators inside the Data Factory browser UI for an integrated experience for data engineers. The Power Query data wrangling feature in Data Factory provides a powerful, easy-to-use pipeline capability to solve your most complex data integration and ETL patterns in a single service.
Learn more
New Stringify data transformation in mapping data flowsMapping data flows adds a new data transformation called Stringify to make it easy to convert complex data types like structs and arrays into string form. These data types then can be sent to structured output destinations.
Learn more
Integration runtime Express virtual network injection for SSIS IR (public preview) The SSIS IR now supports express virtual network injection.
Learn more:
Overview of virtual network injection for SSIS IR
Standard vs. express virtual network injection for SSIS IR
Express virtual network injection for SSIS IR
SecurityAzure Key Vault integration improvementKey Vault integration now has dropdowns so that users can select the secret values in the linked service. This capability increases productivity because users aren't required to type in the secrets, which could result in human error.
Support for user-assigned managed identity in Data FactoryCredential safety is crucial for any enterprise. The Data Factory team is committed to making the data engineering process secure yet simple for data engineers. User-assigned managed identity (preview) is now supported in all connectors and linked services that support Azure AD-based authentication.
Learn more

September 2021


Service categoryService improvementsDetails
Continuous integration and continuous deliveryExpanded CI/CD capabilitiesYou can now create a new Git branch based on any other branch in Data Factory.
Learn more
Data movementAmazon Relational Database Service (RDS) for Oracle sourcesThe Amazon RDS for Oracle sources connector is now available in both Data Factory and Azure Synapse.
Learn more
Amazon RDS for SQL Server sourcesThe Amazon RDS for the SQL Server sources connector is now available in both Data Factory and Azure Synapse.
Learn more
Support parallel copy from Azure Database for PostgreSQLThe Azure Database for PostgreSQL connector now supports parallel copy operations.
Learn more
Data flowUse Data Lake Storage Gen2 to execute pre- and post-processing commandsHadoop Distributed File System pre- and post-processing commands can now be executed by using Data Lake Storage Gen2 sinks in data flows.
Learn more
Edit data flow properties for existing instances of the Azure IR The Azure IR has been updated to allow editing of data flow properties for existing IRs. You can now modify data flow compute properties without needing to create a new Azure IR.
Learn more
TTL setting for Azure Synapse to improve pipeline activities execution startup timeAzure Synapse has added TTL to the Azure IR to enable your data flow pipeline activities to begin execution in seconds, which greatly minimizes the runtime of your data flow pipelines.
Learn more
Integration runtimeData Factory managed virtual network GAYou can now provision the Azure IR as part of a managed virtual network and use private endpoints to securely connect to supported data stores. Data traffic goes through Azure Private Links, which provides secured connectivity to the data source. It also prevents data exfiltration to the public internet.
Learn more
OrchestrationOperationalize and provide SLA for data pipelinesThe new Elapsed Time Pipeline Run metric, combined with Data Factory alerts, empowers data pipeline developers to better deliver SLAs to their customers. Now you can tell us how long a pipeline should run, and we'll notify you proactively when the pipeline runs longer than expected.
Learn more
Fail activity (public preview)The new Fail activity allows you to throw an error in a pipeline intentionally for any reason. For example, you might use the Fail activity if a Lookup activity returns no matching data or a custom activity finishes with an internal error.
Learn more

August 2021


Service categoryService improvementsDetails
Continuous integration and continuous deliveryCI/CD improvements with GitHub support in Azure Government and Azure ChinaWe've added support for GitHub in Azure for US Government and Azure China.
Learn more
Data movementThe Azure Cosmos DB API for MongoDB connector supports versions 3.6 and 4.0 in Data FactoryThe Data Factory Azure Cosmos DB API for MongoDB connector now supports server versions 3.6 and 4.0.
Learn more
Enhance using COPY statement to load data into Azure SynapseThe Data Factory Azure Synapse connector now supports staged copy and copy source with *.* as wildcardFilename for the COPY statement.
Learn more
Data flowREST endpoints are available as source and sink in data flowData flows in Data Factory and Azure Synapse now support REST endpoints as both a source and sink with full support for both JSON and XML payloads.
Learn more
Integration runtimeDiagnostic tool is available for self-hosted IRA diagnostic tool for self-hosted IR is designed to provide a better user experience and help users to find potential issues. The tool runs a series of test scenarios on the self-hosted IR machine. Every scenario has typical health check cases for common issues.
Learn more
OrchestrationCustom event trigger with advanced filtering option GAYou can now create a trigger that responds to a custom topic posted to Azure Event Grid. You can also use advanced filtering to get fine-grain control over what events to respond to.
Learn more

July 2021


Service categoryService improvementsDetails
Data movementGet metadata-driven data ingestion pipelines on the Data Factory Copy Data tool within 10 minutes (public preview)Now you can build large-scale data copy pipelines with a metadata-driven approach on the Copy Data tool (public preview) within 10 minutes.
Learn more
Data flowNew map functions added in data flow transformation functionsA new set of data flow transformation functions enables data engineers to easily generate, read, and update map data types and complex map structures.
Learn more
Integration runtimeFive new regions are available in Data Factory managed virtual network (public preview)Five new regions, China East2, China North2, US Government Arizona, US Government Texas, and US Government Virginia, are available in the Data Factory managed virtual network (public preview).
Developer productivityData Factory home page improvementsThe Data Factory home page has been redesigned with better contrast and reflow capabilities. A few sections are introduced on the home page to help you improve productivity in your data integration journey.
Learn more
New landing page for Data Factory StudioThe landing page for the Data Factory pane in the Azure portal.
Learn more

June 2021


Service categoryService improvementsDetails
Data movementNew user experience with Data Factory Copy Data toolThe redesigned Copy Data tool is now available with improved data ingestion experience.
Learn more
MongoDB and MongoDB Atlas are supported as both source and sinkThis improvement supports copying data between any supported data store and MongoDB or MongoDB Atlas database.
Learn more
Always Encrypted is supported for SQL Database, SQL Managed Instance, and SQL Server connectors as both source and sinkAlways Encrypted is available in Data Factory for SQL Database, SQL Managed Instance, and SQL Server connectors for the Copy activity.
Learn more
Setting custom metadata is supported in Copy activity when sinking to Data Lake Storage Gen2 or Blob StorageWhen you write to Data Lake Storage Gen2 or Blob Storage, the Copy activity supports setting custom metadata or storage of the source file's last modified information as metadata.
Learn more
Data flowSQL Server is now supported as a source and sink in data flowsSQL Server is now supported as a source and sink in data flows. Follow the link for instructions on how to configure your networking by using the Azure IR managed virtual network feature to talk to your SQL Server on-premises and cloud VM-based instances.
Learn more
Dataflow Cluster quick reuse now enabled by default for all new Azure IRsThe popular data flow quick startup reuse feature is now generally available for Data Factory. All new Azure IRs now have quick reuse enabled by default.
Learn more
Power Query (public preview) activityYou can now build complex field mappings to your Power Query sink by using Data Factory data wrangling. The sink is now configured in the pipeline in the Power Query (public preview) activity to accommodate this update.
Learn more
Updated data flows monitoring UI in Data FactoryData Factory has a new update for the monitoring UI to make it easier to view your data flow ETL job executions and quickly identify areas for performance tuning.
Learn more
SQL Server Integration ServicesRun any SQL statements or scripts anywhere in three steps with SSIS in Data FactoryThis post provides three steps to run any SQL statements or scripts anywhere with SSIS in Data Factory.
  1. Prepare your self-hosted IR or SSIS IR.
  2. Prepare an Execute SSIS Package activity in Data Factory pipeline.
  3. Run the Execute SSIS Package activity on your self-hosted IR or SSIS IR.
Learn more

More information