question

MichaelWinick-0809 avatar image
0 Votes"
MichaelWinick-0809 asked MichaelWinick-0809 commented

.NET for Apache Spark: can't read from Event Hubs

I'm attempting to connect my .NET for Apache Spark application to Event Hubs, and following instructions and code described in:
https://github.com/dotnet/spark/issues/325#issuecomment-552536894

Here's my powershell spark-submit:

spark-submit --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.21
--class org.apache.spark.deploy.dotnet.DotnetRunner --master local
bin\Debug\netcoreapp3.1\microsoft-spark-3-0_2.12-1.0.0.jar `
dotnet bin\Debug\netcoreapp3.1\mySparkApp.dll


with the following output/errors:

130838-errors.txt


Any direction would be greatly appreciated.


dotnet-ml-big-data
errors.txt (38.6 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.


         SparkSession spark = SparkSession
             .Builder()
             .AppName("VS Telemetry processing using Spark.NET")
             .GetOrCreate();

         DataFrame events = spark
             .ReadStream()
             .Format("kafka")
             .Options(EventHubConnection.GetEventHubConnectionSettings(10))
             .Load();

         events
             .Select(
                 FromJson(Col("body").Cast("string"), "BlobPath String").Alias("Raw"),
                 Col("properties"),
                 Col("enqueuedTime")
             )
             .SelectExpr("Raw.*", "properties", "enqueuedTime")
             .WithWatermark("enqueuedTime", "60 seconds")
             .CreateOrReplaceTempView("Messages");


StreamingQuery query = spark
.Sql("SELECT * FROM Messages")
.WriteStream()
.OutputMode("append")
.Format("csv")
.Start(@"C:\arcus\mySparkApp\spark_pulled_from_event_hubs.txt");

         Console.WriteLine("Selected from Messages");

         query.AwaitTermination();



0 Votes 0 ·

0 Answers