question

MaheshKumarSSM-8330 avatar image
0 Votes"
MaheshKumarSSM-8330 asked ·

Usage of Python & Spark in Azure Data Factory

I am new to Azure Data Factory. Please help to clarify the following...

  1. Is learning Python & Spark of any help in ADF ?

  2. Are there specific applications/tasks that can only be handled by Spark ?

Thanks.

azure-data-factory
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered ·
  1. Not much. You can write python code or any code to run in spark cluster in Azure databricks then just call this code via notebook from Data factory. ADF is mostly be used to copy the data from various sources, do transformation using Data flow (UI)

  2. Machine learning, streaming data or any data analytics work can be done in spark effectively.


You can try to explore Azure databricks if your focus is more on spark and writing code in python, R, scala or spark sql.


Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav


· 2 ·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Vaibhav,

Thanks for your response !
It is really helpful and it gives me a better idea.

Regards,
Mahesh

0 Votes 0 ·
VaibhavChaudhari avatar image VaibhavChaudhari MaheshKumarSSM-8330 ·

Microsoft has provided learning paths here. Search for data factory, databricks learning path and you can go through them

https://docs.microsoft.com/en-us/learn/browse/

0 Votes 0 ·