Azure Near Realtime Pub/Sub Solution

rvx 1 Reputation point
2021-09-04T08:41:08.85+00:00

Hi All,
I am new to Azure, hence would need some guidance on the below scenario

Below is a conceptual flow
Publisher --> Event hub --> Azure Synapse --> Azure Function --> SQL DW

  1. Publisher: pushes a notification via API, where my system on Azure is one of the Subscriber to this message
  2. Event hub: Can we configure the event hub endpoint at publisher system, so that event hub can persist the message whenever it is published? Or should we need any Azure function sitting between publisher and event hub to capture message and then push to event hub?
  3. Azure Synapse: Message from event hub is required in synapse
  4. Azure function: to apply some business rules and validations
  5. SQL DW final storage of data received from the message

If you have any reference architecture or example to this problem, please do share which would help me in learning.

Thanks in advance

rv

Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
562 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 78,576 Reputation points Microsoft Employee
    2021-09-07T08:58:02.353+00:00

    Hello @rvx ,

    Welcome to the Microsoft Q&A platform.

    From the article- Publisher-Subscriber pattern: It is strongly recommended to use available messaging products and services that support a publish-subscribe model, rather than building your own. In Azure, consider using Service Bus, Event Hubs or Event Grid. Other technologies that can be used for pub/sub messaging include Redis, RabbitMQ, and Apache Kafka.

    The following diagram shows an enterprise integration architecture that uses Service Bus to coordinate workflows, and Event Grid to notify subsystems of events that occur.

    129719-image.png

    For more information, see Enterprise integration on Azure using message queues and events.

    Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second. It facilitates the capture, retention, and replay of telemetry and event stream data. The data can come from many concurrent sources. Event Hubs allows telemetry and event data to be made available to various stream-processing infrastructures and analytics services. It's available either as data streams or bundled event batches. This service provides a single solution that enables rapid data retrieval for real-time processing, and repeated replay of stored raw data. It can capture the streaming data into a file for processing and analysis.

    It has the following characteristics:

    • Low latency
    • Can receive and process millions of events per second
    • At least once delivery of an event

    For more information, see Event Hubs overview..

    This example - Analytics end-to-end with Azure Synapse scenario demonstrates how to use the extensive family of Azure Data Services to build a modern data platform capable of handling the most common data challenges in an organization.

    The solution described in this article combines a range of Azure services that will ingest, store, process, enrich, and serve data and insights from different sources (structured, semi-structured, unstructured, and streaming).

    129862-image.png

    Hope this helps. Do let us know if you any further queries.

    ---------------------------------------------------------------------------

    Please "Accept the answer" if the information helped you. This will help us and others in the community as well.