What is the service or capability in Azure which works like Airflow which will run sequence of flow in particular order i.e. Run pipeline 1 then 2 and then trigger tableau dashboard 3 and once The above three would run in sequence and only then if the p

Question

What is the service or capability in Azure which works like Airflow which will run sequence of flow in particular order i.e. Run pipeline 1 then 2 and then trigger tableau dashboard 3

and once The above three would run in sequence and only then if the previous step has successfully run?

Answer 1

@Mytoast Admin - Thanks for the question and using MS Q&A platform.

Azure offers a service called Azure Data Factory (ADF) that can help you orchestrate and manage data pipelines. ADF allows you to create, schedule, and manage data integration workflows that can move data between various sources and destinations.

In ADF, you can create a pipeline that consists of multiple activities, such as data movement, data transformation, and control activities. You can define dependencies between activities to ensure that they run in a specific order and only when the previous activity has completed successfully.

For example, you can create a pipeline that consists of three activities:

1. Copy data from a source to a destination
2. Transform the data using Azure Databricks
3. Trigger a Tableau dashboard

You can define dependencies between these activities so that activity 2 only runs after activity 1 has completed successfully, and activity 3 only runs after activity 2 has completed successfully.

ADF also provides monitoring and logging capabilities that allow you to track the progress of your pipelines and troubleshoot any issues that may arise.

ADF also

Azure Data Factory offers serverless pipelines for data process orchestration, data movement with 100+ managed connectors, and visual transformations with the mapping data flow.

Azure Data Factory's Workflow Orchestration Manager service is a simple and efficient way to create and manage Apache Airflow environments, enabling you to run data pipelines at scale with ease. Apache Airflow is an open-source platform used to programmatically create, schedule, and monitor complex data workflows. It allows you to define a set of tasks, called operators, that can be combined into directed acyclic graphs (DAGs) to represent data pipelines. Airflow enables you to execute these DAGs on a schedule or in response to an event, monitor the progress of workflows, and provide visibility into the state of each task. It's widely used in data engineering and data science to orchestrate data pipelines, and is known for its flexibility, extensibility, and ease of use.

For more details, refer to Quickstart: Get started with Azure Data Factory and What is Workflow Orchestration Manager.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.