question

rvx-3202 avatar image
0 Votes"
rvx-3202 asked PRADEEPCHEEKATLA-MSFT commented

Azure Near Realtime Pub/Sub Solution

Hi All,
I am new to Azure, hence would need some guidance on the below scenario

Below is a conceptual flow
Publisher --> Event hub --> Azure Synapse --> Azure Function --> SQL DW

  1. Publisher: pushes a notification via API, where my system on Azure is one of the Subscriber to this message

  2. Event hub: Can we configure the event hub endpoint at publisher system, so that event hub can persist the message whenever it is published? Or should we need any Azure function sitting between publisher and event hub to capture message and then push to event hub?

  3. Azure Synapse: Message from event hub is required in synapse

  4. Azure function: to apply some business rules and validations

  5. SQL DW final storage of data received from the message

If you have any reference architecture or example to this problem, please do share which would help me in learning.

Thanks in advance

rv



azure-event-hubs
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @rvx-3202,

Following up to see if the below suggestion was helpful. And, if you have any further query do let us know.


  • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @rvx-3202,

Welcome to the Microsoft Q&A platform.

From the article- Publisher-Subscriber pattern: It is strongly recommended to use available messaging products and services that support a publish-subscribe model, rather than building your own. In Azure, consider using Service Bus, Event Hubs or Event Grid. Other technologies that can be used for pub/sub messaging include Redis, RabbitMQ, and Apache Kafka.

The following diagram shows an enterprise integration architecture that uses Service Bus to coordinate workflows, and Event Grid to notify subsystems of events that occur.

129719-image.png

For more information, see Enterprise integration on Azure using message queues and events.

Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second. It facilitates the capture, retention, and replay of telemetry and event stream data. The data can come from many concurrent sources. Event Hubs allows telemetry and event data to be made available to various stream-processing infrastructures and analytics services. It's available either as data streams or bundled event batches. This service provides a single solution that enables rapid data retrieval for real-time processing, and repeated replay of stored raw data. It can capture the streaming data into a file for processing and analysis.

It has the following characteristics:

  • Low latency

  • Can receive and process millions of events per second

  • At least once delivery of an event

For more information, see Event Hubs overview..

This example - Analytics end-to-end with Azure Synapse scenario demonstrates how to use the extensive family of Azure Data Services to build a modern data platform capable of handling the most common data challenges in an organization.

The solution described in this article combines a range of Azure services that will ingest, store, process, enrich, and serve data and insights from different sources (structured, semi-structured, unstructured, and streaming).

129862-image.png

Hope this helps. Do let us know if you any further queries.


Please "Accept the answer" if the information helped you. This will help us and others in the community as well.


image.png (138.4 KiB)
image.png (375.6 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @rvx-3202,

Just checking in to see if the above answer helped. If this answers your query, do click Accept Answer and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·