Can you please help me with the question i have. I want an idea in creating framework in azure data factory. Below is what i meant by framework.
Generally we will have multiple ADF pipelines in one single ADF pipeline. So, when the main master pipeline fails in production, i want to have a very good control, like in master piepline, which child pipeline failed and if i want to rerun pipeline, it has to skip pipelines which finshed earlier and start from the pipeline which failed and run remaining pipelines in that master pipeline. I am looking for "rerun from point of failure"
Below is what i am planning to do. Is it possible? If not, is there any other idea please?
I will have a Framework database in azure SQLDB and i will connect that database to ADF Pipelines because when pipeline fails it will log all the information into data base like when pipeline started, completed, failed. If pipeline fails, rerun from point of failure i will maintain in stored procedure in database. Pipeline will use that stored procedure and rerun form point of failure.
Is this possible?