If two jobs are Run concurrently to read data from the IOTHUB job getting aborted. even I tried with sequential timings but still, the job is failing it throws the below exception.
Please suggest
Read From IOT Hub to DataBricks
Hi ImranMondal-3977,
Thanks for reaching out. From the error message it seems like an issue related to Azure Event Hubs Spark Connector. Could you please confirm what version of azure-eventhubs-spark is being used? In case if you are using an older version, could you please try the latest version and see if that helps to resolve the issue?
You can get the latest version from Maven repository here: https://mvnrepository.com/artifact/com.microsoft.azure/azure-eventhubs-spark
We look forward to your confirmation.
Thank you
I have verified with EventHubs team and confirmed that this issue could happen if the same consumer group is being used in the concurrent jobs. It has been called out in the connector’s FAQ document that a unique consumer group should be used for each consuming application.
Here is a stackoverflow thread where a similar issue has been discussed with possible resolution: https://stackoverflow.com/questions/54750779/reusing-an-event-hub-stream-for-multiple-queries-in-azure-data-bricks
Hope this helps.
Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.
Just checking in to see if the above suggestion was helpful. If this answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.
Thanks
8 people are following this question.
