question

38577551 avatar image
0 Votes"
38577551 asked KranthiPakala-MSFT commented

How to move Apache Airfow Dags to Azure ?

How to move Apache Airflow Dags to Azure ?

I have Apache Airflow DAGs which are running on GCP and have python code in it and would like to know how can I move this to Azure .

Is there a ETL tool which will convert DAG to some other ETL program?

Should the solution be shifted as is or must be re-architected if so what would be tool of choice?

azure-data-factorysql-server-integration-servicesazure-databricks
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @38577551,

Welcome to Microsoft Q&A forum and thanks for posting your query.

In Azure, ADF (Azure Data Factory) is the cloud-based ETL and data integration service permitting data-driven work processes for arranging data development and changing data at scale. With Azure Data Factory, pipelines (schedule data-driven workflows) can ingest data from unique data stores. You can assemble complex ETL measures that change data outwardly with information streams or by utilizing register administrations like Azure HDInsight Hadoop, Azure Databricks, and Azure SQL Database.

As per your requirement, I believe the service you are looking for might be Azure Databricks as Airflow provides tight integration between Azure Databricks and Airflow. The Airflow Azure Databricks integration lets you take advantage of the optimized Spark engine offered by Azure Databricks with the scheduling features of Airflow.

For more details about how to install the Airflow Azure Databricks integration, configuring a Databricks connection, please refer to this article : Managing dependencies in data pipelines

Also please refer to this article and see if it can be of help for your requirement : Deploying Apache Airflow in Azure to build and run data pipelines

Hope this info helps.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @38577551,

Just checking in to see if the above suggestion was helpful. If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

0 Votes 0 ·

Hi @38577551,

We still have not heard back from you. Just wanted to check if the above suggestion was helpful? If it answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

0 Votes 0 ·