Can Azure Purview ingest Apache Airflow Dags to show technical lineage of a column which undergoes transformation ?
If not what is recommended by Azure Purview team to ingest the Apache Airflow Dags?
Can Azure Purview ingest Apache Airflow Dags to show technical lineage of a column which undergoes transformation ?
If not what is recommended by Azure Purview team to ingest the Apache Airflow Dags?
Hi @DataArtist,
Thanks for using Microsoft Q&A forum and posting your query.
Unfortunately Apache Airflow Dags is not supported with Azure Purview.
I would recommend you to please log a feature request in Azure purview feedback forum: https://feedback.azure.com/forums/932437-azure-purview. All the feedback shared in this forum are actively monitored and reviewed by Azure Purview team. Also please do share the feedback link here once it is posted so that others with similar idea can upvote and comment on it.
I will reach out to product team to check if there is any alternate/workaround for this and get back to you as soon as I have a response from the team.
Hope this info helps
Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
Submitted the request
Apache Airflow Dags is not a data store , it is a ETL program which runs Python code .
Hi @dataartist, Thank you for your detailed feedback and apologies for not being clear on my initial response, I will update it accordingly.
As per my conversation with Product team, here is a stackoverflow thread which indicates that while the integration will work from an Apache Atlas perspective, it may not work similarly in Purview since I am not sure if we expose the Apache Atlas UI (port 21000) via Purview.
I will share this feedback link with Azure Purview team and will let you know if I have any updates from the team.
Once again thank you so much for providing a detailed feedback verbatim and helping Microsoft build better products. :)
2 people are following this question.