I am new to Azure Data Factory. Please help to clarify the following...
Is learning Python & Spark of any help in ADF ?
Are there specific applications/tasks that can only be handled by Spark ?
Not much. You can write python code or any code to run in spark cluster in Azure databricks then just call this code via notebook from Data factory. ADF is mostly be used to copy the data from various sources, do transformation using Data flow (UI)
Machine learning, streaming data or any data analytics work can be done in spark effectively.
You can try to explore Azure databricks if your focus is more on spark and writing code in python, R, scala or spark sql.
Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav
6 people are following this question.