question

narensrinivas-5062 avatar image
0 Votes"
narensrinivas-5062 asked

How to Execute Dotnet apache spark using docker image?

  1. I have created the docker file with Dotnet SDK

  2. Copied my debug folder of my console

  3. Create the package(Folder) which have all the required software for running dotnet pyspark application and copied this also to my image file

Find the Docker image details below

FROM mcr.microsoft.com/dotnet/sdk:5.0

WORKDIR /app/Debug

COPY ./Debug .

WORKDIR /app/DotnetSpark.Package

COPY ./DotnetSpark.Package .

ENV HADOOP_HOME=C:/app/DotnetSpark.Package/hadoop
ENV SPARK_HOME=C:/app/DotnetSpark.Package/spark-3.0.0-bin-hadoop2.7
ENV DOTNET_WORKER_DIR=C:/app/DotnetSpark.Package/Microsoft.Spark.Worker-1.0.0
ENV DOTNET_ASSEMBLY_SEARCH_PATHS=C:/app/Debug
ENV JAVA_HOME=C:/app/DotnetSpark.Package/java
ENV DOTNET="C:/Program Files/dotnet"
ENV TEMP=C:/Windows/Temp
ENV TMP=C:/Windows/Temp

PATH=${HADOOP_HOME}/bin;${SPARK_HOME}/bin;${DOTNET_WORKER_DIR};${JAVA_HOME}/bin;${DOTNET};C:/Windows/system32;C:/Windows;${TEMP};${TMP}

ENTRYPOINT ["cmd.exe"]

After running this docker, cmd prompt is getting open, then i tried to execute the below command

spark-submit --class org.apache.spark.deploy.dotnet.DotnetRunner --master local "c:\app\debug
\packages\Microsoft.Spark.1.0.0\jars\microsoft-spark-3-0_2.12-1.0.0.jar" "c:\app\debug\spark_app.exe"

But this command is not throwing any exception or result, i'm getting result like below image

Can any one help to resolve my problem or suggest a way to run dotnet pyspark using docker?

99462-image.png




dotnet-addotnet-ml-big-data
image.png (21.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

0 Answers