Reusing ADLS paths on failed spark streaming.

Question

question

PeconcilloPaulRyan-6090 asked Mar 4, '22 HimanshuSinha-MSFT commented Mar 24, '22

Reusing ADLS paths on failed spark streaming.

Hi,

We have an Event hub which is used to ingest data from an external sending application. We are reading data from the Eventhub using Azure databricks using Pyspark and stream it on a ADLS sink. But in the event of a failure of the streaming for some reason when I tried to re-run my spark streaming on Azure databricks I got this error "java.lang.NoSuchMethodError: org.apache.spark.sql.eventhubs.EventHubsSource$$anon$1.parseVersion(Ljava/lang/String;I)I" . To run my spark streaming again, I need to delete my sink in the ADLS and restart, its kind of annoying and you should not delete all the events generated after it failed.

azure-databricks azure-event-hubs dotnet-ml-big-data

· 4

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT · Mar 08 at 05:51 PM

Hello @PeconcilloPaulRyan-6090 ,
Thanks for the ask and using Microsoft Q&A platform.
Can you please sharing the code snippet ? That will help us look into this issue further .
Thanks
Himanshu

0 ·

PeconcilloPaulRyan-6090 HimanshuSinha-MSFT · Mar 09 at 10:37 AM

iHi Below is my code snippet

Read Event Hub's stream
conf = {}
cnxString = event_hub_endpoint
conf["eventhubs.connectionString"] = sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(cnxString)

read_from_ehub_df = (
spark.readStream
.format("eventhubs")
.options(**conf)
.load()
)

Decode body column
decoded_from_ehub_df = (
read_from_ehub_df .withColumn("body", read_df["body"].cast("string"))
.select(col("sequenceNumber"),col("body"),col("enqueuedTime"))
)

Stream from EventHun to ADLSg2
stream_query = (
decoded_from_ehub_df.writeStream
.outputMode("append")
.format("parquet")
.option("checkpointLocation",checkpointPath)
.option("path",outputPath)
.start()
)

0 ·

HimanshuSinha-MSFT PeconcilloPaulRyan-6090 · Mar 23 at 06:32 PM

Hello @PeconcilloPaulRyan-6090,
Thanks for the patience , unfortunately we are not getting the kind of response from the team here . if you have a support plan you may file a support ticket, else could you please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.
Subscription ID:
Subject : Attn Himanshu
Please let me know once you have done the same.
Thanks
Himanshu

0 ·

Show more comments

question