question

ImranMondal-3977 avatar image
1 Vote"
ImranMondal-3977 asked KranthiPakala-MSFT commented

Read From IOT Hub to DataBricks

If two jobs are Run concurrently to read data from the IOTHUB job getting aborted. even I tried with sequential timings but still, the job is failing it throws the below exception.

84456-error.jpg



Please suggest

azure-databricksazure-iot-hub
error.jpg (129.0 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi ImranMondal-3977,

Thanks for reaching out. From the error message it seems like an issue related to Azure Event Hubs Spark Connector. Could you please confirm what version of azure-eventhubs-spark is being used? In case if you are using an older version, could you please try the latest version and see if that helps to resolve the issue?

You can get the latest version from Maven repository here: https://mvnrepository.com/artifact/com.microsoft.azure/azure-eventhubs-spark

We look forward to your confirmation.

Thank you

0 Votes 0 ·

1 Answer

KranthiPakala-MSFT avatar image
2 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @ImranMondal-3977,

I have verified with EventHubs team and confirmed that this issue could happen if the same consumer group is being used in the concurrent jobs. It has been called out in the connector’s FAQ document that a unique consumer group should be used for each consuming application.

84595-image.png

Here is a stackoverflow thread where a similar issue has been discussed with possible resolution: https://stackoverflow.com/questions/54750779/reusing-an-event-hub-stream-for-multiple-queries-in-azure-data-bricks

Hope this helps.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.




image.png (55.6 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @ImranMondal-3977,

Just checking in to see if the above suggestion was helpful. If this answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thanks

0 Votes 0 ·