Eventhub to Service bus (Structured streaming)

Chand, Anupam SBOBNG-ITA/RX 451

We are currently using Databricks for structured stream processing from Eventhub. Our aim is to push these events into a Service bus which can then be picked up by functions for the heavy processing.
Someone mentioned that Structured streaming to Service bus is not possible. However, can't we write the individual events or batched up events to Service bus? Is there anything we need to keep in mind before trying that?

KranthiPakala-MSFT 46,422 Reputation points Microsoft Employee

2021-01-26T23:48:21.983+00:00

Hi @Chand, Anupam SBOBNG-ITA/RX ,

Welcome to Microsoft Q&A forum and thanks for your query.

Your query seems similar to : Spark Structured Streaming Azure Service Bus
For additional help on this requirement, I have reached out to internal team and will get back to you as soon we have an update from the team.

We appreciate your patience.

Thank you

Accepted answer

MayankBargali-MSFT 68,391 Reputation points

2021-01-27T06:06:25.243+00:00

Hi @Chand, Anupam SBOBNG-ITA/RX

Service Bus is not designed for spark structured streaming and the better option would be using Eventhub for structured streaming. But if still, you want to use a service bus, then you need to create your own receiver like this one as mentioned in the previous discussion which @KranthiPakala-MSFT has shared.

Feel free to get back to me if you have any queries or concerns.
Please sign in to rate this answer.
Chand, Anupam SBOBNG-ITA/RX 451 Reputation points

2021-01-27T06:27:05.197+00:00

Thanks @MayankBargali-MSFT ,

Our overall architecture is already using Eventhubs for ingestion and a databricks notebook which will curate and filter the data.

We were proposing to have the individual events go into a servicebus (to ensure loosely coupled architecture) and then have functions kickoff via the service bus messages and do some processing on the data. Ideally we would have liked individual events to be in the service bus. Can this still be achieved using the approach you have mentioned? Or is there a way to split the stream into individual events before feeding into the Service bus?

Thanks and Regards,
Anupam

MayankBargali-MSFT 68,391 Reputation points

2021-01-27T06:40:22.487+00:00

@Chand, Anupam SBOBNG-ITA/RX As you will be writing your own code so I think it should be possible. Client application need to have their own logic to split the stream into supported service bus message size
Sign in to comment

Eventhub to Service bus (Structured streaming)

0 additional answers