question

nikhilsharma3-1623 avatar image
0 Votes"
nikhilsharma3-1623 asked HimanshuSinha-MSFT edited

How to run .NET Spark jobs on Databricks from Azure Data Factory?

In azure data factory, you have a Databricks Acvitiy. This activity supports running python, jar and notebooks. And These notebooks may be written in scala, python, java, and R but not c#/.net.

Is there inherent or direct support where I can write my .NET spark code and run it on Databricks from Data Factory?



azure-data-factoryazure-databricks
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

VaibhavChaudhari avatar image
0 Votes"
VaibhavChaudhari answered KranthiPakala-MSFT converted comment to answer

I'm not sure if it is possible in Azure Databricks and Data factory.

But this might be possible in Azure synapse notebook where you can write .Net for spark code and may be schedule synapse notebook in Synapse pipeline.

See this video : Scaling-NET-for-Apache-Spark-processing-jobs-with-Azure-Synapse


===============================================
If the response helped, do "Accept Answer" and upvote it -- Vaibhav

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi Vaibhav,

Thanks for your reply. Are you sure that there isn't any direct support for invoking .NET spark on Azure Databricks activity with Azure Data factory

0 Votes 0 ·
HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered HimanshuSinha-MSFT edited

Hello ,

Thanks for asking the question and using the forum .

We have something in preview at this time . We don't have a timeline when it will GA .
https://docs.microsoft.com/en-us/dotnet/spark/tutorials/databricks-deployment.
Thanks
Himanshu

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.