question

AsifulHaqueLatifNobel-4615 avatar image
1 Vote"
AsifulHaqueLatifNobel-4615 asked DeekshaThakur-9499 commented

Use-case of Azure Event Hub Consumer Group

Can someone explain what is the point of having multiple consumer groups vs single consumer group in an Event Hub instance? I checked the Microsoft Docs on this topic, but did not get an answer. It says that consumer groups let multiple applications to each have a separate view. But it does not show anywhere how that is done.

For example, imagine a scenario where there are 4 applications (all are single instance) consuming from the same Event Hub instance. All 4 apps will be able to read all message from the Event Hub regardless of whether the apps are using one consumer group or four, right?


azure-event-hubs
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

HimanshuSinha-MSFT avatar image
0 Votes"
HimanshuSinha-MSFT answered AsifulHaqueLatifNobel-4615 commented

Hello @AsifulHaqueLatifNobel-4615 ,
Thanks for the ask and using the Microsoft Q&A platform . I think you are missing the point you can write the events to a particular partition(s) and label a consumer group to that . The below diagram shoukld help .
107821-image.png

This link should also make things more clear .

Please do let me know how it goes .
Thanks
Himanshu
Please do consider clicking on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members



image.png (75.2 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for replying. I cannot see the image that you attached, but was able to get to the GitHub link. I have read that already, but it did not answer my question about the unique view for a consumer.

Because I know I can add a property in a published event to write it to a particular partition. But what does that have to do anything with consumer group? Also the GitHub link says that conceptually consumer group is a label that identifies one or more event consumers as a set. That makes sense, but is it only for that? Grouping of consumer apps? Another thing is I don't get how consumer group lets an app read and process events from an Event Hub independently from other consumer app. Because an app reads events from Event Hub and checkpoints it using Storage Account, right? After reading an event, it does not get cleared from Event Hub until the it is older than retention period as far as I know.

Then where is the unique view for a consumer group?

0 Votes 0 ·
SerkantKaraca-MSFT avatar image
0 Votes"
SerkantKaraca-MSFT answered DeekshaThakur-9499 commented

I think it is easier to describe it with an example. Let's say there is an eventhub providing some purchase order notifications and you need to process messages from that in 2 different pipelines. One for telemetry that needs to consume messages near real time as they arrive; this may well be a job to send an email notification to customers when order state changes. Then, 2nd pipeline where processor collects telemetry like data which can run every once in a while (batch processing) and write statistical data to some downstream system. As you can see, in this scenario you need to track checkpoints separately, one for notification processor and one for telemetry processor. Each processor needs to know the position of the last message they processed so they can resume the process in case of a failure or system restart. Consumer groups come into play to isolate consumers in such scenarios where consumer behavior or code starts differentiating. In other words each group of consumer runs with its own consumer-group so they don't step on each other's toe.

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

That sounds good. However, doesn't checkpointing actually save the checkpoint information in a storage account as a blob? In that case, would I not be able to have two consumer applications in same consumer group with separate storage accounts consume from the same event hub instance? Would that still cause an issue with consumers getting in each other's way?

I am actually trying to understand if having a consumer group per application has any performance benefit or is it just to logically group consumer applications?

1 Vote 1 ·
DeekshaThakur-9499 avatar image DeekshaThakur-9499 AsifulHaqueLatifNobel-4615 ·

I am looking for clarification on this as well.

0 Votes 0 ·