question

86097587 avatar image
0 Votes"
86097587 asked PRADEEPCHEEKATLA-MSFT commented

Can I specify command line arguments when deploying .NET for Apache Spark with Spark Submit in Azure Databricks?

I've modified "Get started in 10 minutes" to allow you to specify a file name.

>spark-submit ^
--class org.apache.spark.deploy.dotnet.DotnetRunner ^
--master local ^
microsoft-spark-2-4_2.11-1.0.0.jar ^
dotnet MySparkApp.dll input.txt

How do I get it to work this with Azure Databricks?

["--class","org.apache.spark.deploy.dotnet.DotnetRunner","/dbfs/spark-dotnet/microsoft-spark-2-4_2.11-1.0.0.jar","/dbfs/spark-dotnet/publish.zip","MySparkApp"]

dotnet-csharpazure-databricksdotnet-ml-big-data
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @86097587,

Thanks for the question and using the MS Q&A platform.

You can run your .NET for Apache Spark jobs on Databricks clusters, but it is not available out-of-the-box. There are two ways to deploy your .NET for Apache Spark job to Databricks: spark-submit and Set Jar.

For more details, refer to Submit a .NET for Apache Spark job to Databricks.

Hope this helps. Do let us know if you any further queries.


  • Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

  • Want a reminder to come back and check responses? Here is how to subscribe to a notification.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thank you very much for this.

I was able to run the .NET for Apache Spark job by specifying the command line arguments as follows.

["--class","org.apache.spark.deploy.dotnet.DotnetRunner","/dbfs/spark-dotnet/microsoft-spark-2-4_2.11-1.0.0.jar","/dbfs/spark-dotnet/publish.zip","MySparkApp", "input.txt"]

Also found out from the error message below that even if input.txt is included in publish.zip, it needs to be uploaded separately as dbfs:/input.txt.

Microsoft.Spark.JvmException: org.apache.spark.sql.AnalysisException: Path does not exist: dbfs:/input.txt;

0 Votes 0 ·

Hello @ReddyHaripriya-9550,

Glad to know it helped.

0 Votes 0 ·