February 2018
Volume 33 Number 2
[Azure]
Event-Driven Architecture in the Cloud with Azure Event Grid
By David Barkol
It’s an exciting time to be a cloud architect. The pace of innovation has brought to the forefront a set of new challenges and technologies that are reshaping the way solutions are designed. Benefitting from this growth are the developers and architects who can choose from an abundance of services and options.
As developers go through the exercise of decomposing their architectures to take advantage of new services like Azure Functions, Logic Apps and others, familiar obstacles surface. In many cases, we find ourselves once again piecing together the “glue” that allows these services to work in concert. The recent launch of Azure Event Grid aims to solve this challenge by providing a first-class event routing service in the cloud that’s fully managed, scalable and extremely flexible.
In this article, I’ll explore the flexibility of Azure Event Grid and show how it can be used to resolve familiar challenges in enterprise applications. I also encourage you to check out the Event Grid general availability announcement at aka.ms/egblog.
The Reversal of Dependencies
The notion of using events in a solution or application isn’t new. In fact, event-driven programming has successfully utilized the concept for quite some time. Pub/Sub queues and GUIs are just a few of the examples that leverage this idea of reacting to events that happen in an organization, application or system.
One of the core tenets of an event-driven architecture is to reverse the dependencies that existing services may have with each other. Figure 1 shows an example of a set of processes that rely on each other to communicate and support a Human Resources (HR) department.
.png "Services That Contain Logic About Other Services")
Figure 1 Services That Contain Logic About Other Services
For this design to work, each service must contain some basic logic about the others to communicate. These dependencies create challenges, not only in terms of scale, but also due to the fact that this logic is scattered across the architecture. Over time, as these types of solutions expand, they become difficult to maintain and increasingly brittle as more changes and dependencies are introduced.
As an alternative, the concept behind an event-driven design removes these dependencies by promoting the idea of an event as something that’s a first-class citizen in the architecture. This important consideration allows many other systems the ability to leverage a centralized service without the burden of dependencies and logic dispersed throughout the application. Figure 2 highlights the reversal of dependencies for the HR department solution by introducing this principal concept.
.png "A Centralized Service That Reverses Dependencies Between the Other Services")
Figure 2 A Centralized Service That Reverses Dependencies Between the Other Services
This key service is what the rest of the article is all about. I’ll be exploring Azure Event Grid and how it can be used to support the next generation of solutions.
Introducing Azure Event Grid
Azure Event Grid is a new, fully managed service that supports the routing of events by utilizing a publisher-subscriber model. At its core, Event Grid is an event routing service that manages the routing and delivery of events from numerous sources and subscribers. Figure 3, taken from the Event Grid overview documentation (bit.ly/2qhaj9q), illustrates several of the publishers and handlers that can be used with Event Grid today.
.png "Azure Event Grid Overview")
Figure 3 Azure Event Grid Overview
An event is created by a publisher such as a Blob Storage account, Event Hubs or even an Azure subscription. As events occur, they’re published to an endpoint called a topic that the Event Grid service manages to digest all incoming messages. The list of services on Azure that integrate with Event Grid is growing, with many more on the horizon.
Event publishers aren’t limited to services on Azure. In fact, a very common use case includes events that originate from custom applications or systems that can run from anywhere. This includes applications that are hosted on-premises, in a datacenter, or even on other clouds. These types of publishers are referred to as Custom Topics. If they can post an HTTP request to the Event Grid service, then they’re candidates for sending events.
Event handlers include several services on Azure, as well. These showcase some of the emerging serverless technologies on Azure, such as Functions and Logic Apps. In addition to Azure Automation, another type of event handler could be any HTTP callback, also referred to as a WebHook. Handlers are registered with Event Grid by creating an event subscription. If the event handler endpoint is publicly accessible and encrypted by Transport Layer Security, then messages can be pushed to it from Event Grid.
Unlike many other Azure services, there’s no Event Grid namespace that needs to be provisioned or managed. Topics for native Azure resources are built in and completely transparent to users while custom topics are provisioned ad hoc and exist in a resource group. Event subscriptions are simply associated with a topic. This model simplifies management of topics as subscriptions and makes Event Grid highly multi-tenant, allowing for massive scale out.
Azure Event Grid is agnostic to any language or platform. While it integrates natively with Azure services, it can just as easily be leveraged by anything that supports the HTTP protocol, which makes it a very clever and innovative service.
Events or Commands
Before I dive into some code and build out a solution that highlights some of these features, lets distinguish between an event and a command. The distinction can sometimes be subtle, but it’s important to comprehend when designing systems that rely on messages.
When a message is sent a specific action or response, it’s most likely a command. For example, if an employee is promoted within an organization and a message is sent to instruct his new manager to fill out a form, then it carries with it a specific purpose or intent. Because the sender of the message has an expectation, and in some cases, might even expect a response, we can categorize this as a command.
If a message is published without any knowledge or expectations of how it will be handled, then it’s deemed to be an event. Let’s say that within an organization, the same employee has requested to change their mailing address. Because this action might be interesting to many systems in the organization but doesn’t require the publisher to specifically be aware of any of them, it’s a message that doesn’t define any intent. In this case, the publisher is simply notifying any interested parties that an event has occurred. It’s an event, and clearly an occurrence of something that’s a viable option for a service like Event Grid.
I can spend a lot more time discussing these distinctions and in addition how to select the appropriate messaging service on Azure, however, that’s out of the scope of this article. I encourage you to read this insightful post from Clemens Vasters on the topic: bit.ly/2CH3sbQ.
A Human Resources Scenario
The best way to get a deeper understanding of Event Grid is by writing code that leverages its capabilities. In this article, I’ll look at a few events that originate from a fictitious HR application. I’ll publish events to a custom topic and then subscribe to the events using several different handlers.
To keep things simple, I’ll be implementing two types of events from the HR system—when an employee is added to an organization and when an employee is removed. These events are close enough in nature that it will provide options that showcase how to filter and handle events in diverse ways. A visual representation of the solution is illustrated in Figure 4.
.png "A Sample Solution")
Figure 4 A Sample Solution
At a high-level, the solution consists of several key components that I will build in this article. Let’s explore them here.
Employee Events will be an Event Grid Topic to which the HR application can send messages. This will include events for new and removed employees in the organization. Each message will contain information about the employee, her department and the type of event.
New Employee Welcome will be a Logic App that subscribes to messages for new employees in the organization. It will ultimately send a welcome email to the new employee.
New Employee Equipment Order is an Azure Function that will subscribe to events for new employees in the Engineering department. It will then create a message in a queue for additional processing.
Employee Records is a custom Web site built on ASP.NET Core that will expose a Web API for receiving messages when employees leave the organization.
Creating a Custom Topic
To get started, I’ll need to create a few basic resources in Azure. You can either launch the Azure Cloud Shell from the portal or use the command-line interface (CLI) locally. Details on how to use the Cloud Shell can be found at bit.ly/2CsFtQB. If you haven’t used Cloud Shell before, I highly recommend it.
The first thing I’m going to do is create a resource group to manage and encapsulate the Azure resources:
az group create --name <resource-group-name> --location <location>
After the group is created, an Event Grid Topic is provisioned. This will provide an endpoint to publish custom events from the HR application. The name of the topic must be unique to the region, as it will be a publicly accessible service on Azure. The location must also be in a region that has the Event Grid service available. I usually go with the westus2 location or refer to the list of services provided in each Azure region (see bit.ly/2DU15ln).
az eventgrid topic create --name <topic-name> \
--location <location> \
--resource-group <resource group name>
After executing the command to create the topic, details about the resource will be returned. The output will look similar to, but not exactly like, the code here:
{
"endpoint": "https://<topic name>.westus2-1.eventgrid.azure.net/api/events",
"id": "/subscriptions/xxxx-xxx-xx-xxx-xx/resourceGroups/eventgridsolution-rg/providers/Microsoft.EventGrid/topics/<topic name>",
"location": "westus2",
"name": "<topic name>",
"provisioningState": "Succeeded",
"resourceGroup": "eventgridsolution-rg",
"tags": null,
"type": "Microsoft.EventGrid/topics"
}
Make note of the endpoint value—it will be used later when publishing events. You’ll also need one of the two access keys that were generated for authorization. To retrieve the keys, you can list the ones associated with the topic. You can and should cycle and regenerate these keys as a security measure—just like you would with other services on Azure.
az eventgrid topic key list --name <topic-name> --resource-group <resource-group-name>
If your preference is to work within the Azure Portal, you can create and view all these options and settings there, as well.
Publishing an Event
Before sending the first event, you need to understand the event schema that’s expected by the topic. Each event, regardless of if the publisher is an Azure resource or custom application, will adhere to the structure outlined in the following code (a helpful reference for the event schema, as well as some examples, can be found at bit.ly/2CG8oxI):
[
{
"topic": string,
"subject": string,
"id": string,
"eventType": string,
"eventTime": string,
"data":{
object-unique-to-each-publisher
}
}
]
The first thing to point out is that events are sent in an array. This is purposely done to provide the ability to send multiple events within a request. Events can be sent in batches, which reduces network chattiness while supporting scenarios where connectivity isn’t always available.
The first event I want to publish is for when a new employee joins an organization. The payload for this event may resemble the contents of Figure 5.
Figure 5 Employee Added Event
[{
"id": "30934",
"eventType": "employeeAdded",
"subject": "department/engineering",
"eventTime": "2017-12-14T10:10:20+00:00",
"data":{
"employeeId": "14",
"employeeName": "Nigel Tufnel",
"employeeEmail": "nigel@contoso.com",
"manager": "itmanager@contoso.com",
"managerId": "4"
}
}]
Later in the article I’ll use the same structure, with some differences in the values, as when an employee leaves the organization. The key properties in this event are as follows:
eventType is a value used to uniquely identify the published event type. This property can be used by handlers wishing to subscribe only to specific event types, rather than all types.
subject is a value, like eventType, that’s available to provide additional context about the event, with the option of also providing an additional filter to subscribers. I’ll leverage both eventType and subject soon when I create the subscriptions. Subject and eventType give context to the event.
data is a publisher-defined bucket that’s simply an object that can contain one or more properties. Publishers assign relevant information about the event itself inside this property. For example, the Azure Blob storage event includes details about the created or deleted blob such as URL and content type.
To publish the event, I use Postman (or a similar tool) to simulate the message coming from the HR application to the endpoint address mentioned earlier. For authorization, I add an item in the header called aeg-sas-key—it’s value is one of the access keys generated when the topic is created. The body of the request will contain the payload mentioned in Figure 5.
Because there aren’t any subscribers, there isn’t anything to observe yet. The next step will be to see this in action by creating a few event handlers for Event Grid to push the events to.
Handling Events with an Azure Function
Now comes the fun part of subscribing to events. Our first handler will be an Azure Function. To learn the basics of creating a Function, see bit.ly/2A6pFgu. For this situation, I want to specifically subscribe to events for recently added employees. Additionally, and just as important, this handler must only be invoked for employees that belong to theengineering department.
Most examples walk through the creation of a Function using the Azure Portal—which is super-easy and quick. I’d like to show you how to do this locally, from Visual Studio. This will pave the way for more production-ready code. I’ll also use a utility called ngrok (see ngrok.com) to support local debugging with Event Grid.
If you’d like to follow along, you’ll need ngrok, as well as an up-to-date version of Visual Studio (I used version 15.5.2 for this article). Let’s start by creating a new project and selecting Azure Functions from the Cloud templates. In the New Project dialog box, select the HTTP trigger option and keep the default values.
Update the code for the function to reflect what’s represented in Figure 6. Feel free to rename the file to reflect the function name.
Figure 6 Implementation for the New Employee Event Handler>
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Extensions.Http;
using Microsoft.Azure.WebJobs.Host;
using Newtonsoft.Json;
namespace NewEmployeeApp
{
public static class NewEmployeeHandler
{
public class GridEvent<T> where T : class
{
public string Id { get; set; }
public string EventType { get; set; }
public string Subject { get; set; }
public DateTime EventTime { get; set; }
public T Data { get; set; }
public string Topic { get; set; }
}
[FunctionName("newemployeehandler")]
public static async Task<HttpResponseMessage> Run(
[HttpTrigger(AuthorizationLevel.Function, "post", Route = null)]
HttpRequestMessage req,
TraceWriter log)
{
log.Info("New Employee Handler Triggered");
// Retrieve the contents of the request and
// deserialize it into a grid event object.
var jsonContent = await req.Content.ReadAsStringAsync();
var gridEvent =
JsonConvert.DeserializeObject<List<GridEvent<Dictionary<string,
string>>>>(jsonContent)
?.SingleOrDefault();
// Check to see if the event is available and
// return an error response if its missing.
if (gridEvent == null)
{
return req.CreateErrorResponse(HttpStatusCode.BadRequest,
$@"Missing event details");
}
// Check the header to identify the type of request
// from Event Grid. A subscription validation request
// must echo back the validation code.
var gridEventType = req.Headers.GetValues("Aeg-Event-Type").
FirstOrDefault();
if (gridEventType == "SubscriptionValidation")
{
var code = gridEvent.Data["validationCode"];
return req.CreateResponse(HttpStatusCode.OK,
new { validationResponse = code });
}
else if (gridEventType == "Notification")
{
// Pseudo code: place message into a queue
// for further processing.
return req.CreateResponse(HttpStatusCode.OK);
}
else
{
return req.CreateErrorResponse(HttpStatusCode.BadRequest,
$@"Unknown request type");
}
}
}
}
There is some important code to review here. At the very beginning is a class called GridEvent that’s intended to reflect the payload and event schema from Event Grid. Ideally, I would place this class in a common library so it can be reused. For this example, it’s used to deserialize the contents of the request into a strongly typed object.
Event Grid will send to its subscribers two types of requests—SubscriptionValidation and Notification—that you can identify by inspecting a value from the header. The validation request is important to ensure that all subscribers are added explicitly. All I must do here is echo back the validation code to acknowledge that I can receive messages:
var code = gridEvent.Data["validationCode"];
return req.CreateResponse(HttpStatusCode.OK,
new { validationResponse = code });
Validation requests can also be identified by their event type: Microsoft.EventGrid.SubscriptionValidationEvent. If the event type is Notification, then I proceed with the implementation of the business logic. This defensive programming approach is highly recommended when exposing endpoints to other services.
Functions that are hosted in Azure, and are referenced with the azurewebsites.net domain, do not require the subscription validation logic. Instead, they’re whitelisted by Event Grid along with several other services such as Logic Apps and callbacks from Azure Automation run books. Because I plan to test locally, I need to echo back the validation code for Event Grid to acknowledge the function as a valid endpoint.
Eventually the Event Grid runtime SDK will handle much of this setup, from deserializing events and creating strongly typed Event Grid objects, to validating endpoints automatically. As of this writing, the updated runtime SDKs were not yet available.
Local Function Testing
Let’s start the function from Visual Studio so that it runs locally on port 7071. Once it’s running, open a command prompt and use ngrok to create a secure tunnel:
ngrok http -host-header=localhost 7071
I’ll get back an HTTPS address from ngrok to use as the subscriber endpoint. The address should look something like https://d69f6bed.ngrok.io, but with a different subdomain each time the ngrok command is executed. Append the route of our function to the URL so that it resembles something like https://<generated-value>.ngrok.io/api/newemployeehandler. This will be the endpoint address for the event subscription.
With the function running and the secure tunnel in place, I can now create the event subscription from the CLI or Azure Cloud Shell:
az eventgrid event-subscription create --name <event-subscription-name> \
--resource-group <resource group name> \
--topic-name <topic name> \
--subject-ends-with engineering \
--included-event-type employeeAdded \
--endpoint <function endpoint>
Optionally, I can add an event subscription from the portal by filling out the dialog, as shown in Figure 7.
.png "Creating an Event Subscription from the Portal")
Figure 7 Creating an Event Subscription from the Portal
I want to call out several important arguments in the creation of the event subscription.
subject-begins-with (Prefix Filter) is an optional argument that filters based on the prefix of the subject field in the events. It is a literal string match. Wildcard characters and regular expressions are not supported.
subject-ends-with (Suffix Filter) is an optional argument based on a suffix to filter events. Wildcard characters and regular expressions are not supported.
included-event-type (Event Types) is an optional list of event types to subscribe to. Each type is separated by a space.
Now I can return to the publishing event example earlier in the article to ensure that events are flowing through from Postman, to Event Grid, and ultimately to the local function. Feel free to change the values in the request to validate the filters are working as expected.
Handling Events: Logic App and WebHook
The next event subscription is a Logic App. Like the Azure Function example, it’s only interested in the added employee event type. It won’t leverage the prefix or suffix filters, because I want to send a message to employees from all departments. The completed version of the Logic App is shown in Figure 8.
.png "A Logic App That Welcomes New Employees")
Figure 8 A Logic App That Welcomes New Employees
The Logic App begins with an Event Grid trigger. Selecting Microsoft.EventGrid.topics as the resource type will allow me to pick from the custom topics in the subscription.
The Parse JSON action will be helpful in accessing the properties within the Data object. I’ll use this sample payload to generate the schema:
{
"id": "40000",
"eventType": "employeeAdded",
"subject": "department/finance",
"eventTime": "2017-12-20T10:10:20+00:00",
"data":{
"employeeId": "24",
"employeeName": "David St. Hubbins",
"employeeEmail": "david@contoso.com",
"manager": "finance@contoso.com",
"managerId": "10"
}
}
Next, the filter for the event type must be done with a condition action. This is a slight departure from how the event subscription was created with the Function because there isn’t a way to select this option in the Event Grid trigger.
The last step sends an email to the employee. It uses the properties that were retrieved from the second step to populate the recipient address and subject fields of the email. To test the Logic App, click Run from the designer and send a message to the endpoint like before.
The last event subscription is a basic HTTP callback, or WebHook. I’ll update an existing ASP.NET Core application with a Web API for incoming events. The code for the WebHook will be very similar to the Azure Function I wrote earlier. Some subtle differences include the way header values are retrieved to inspect the request type, as seen in Figure 9.
Figure 9 A Web API Controller That Receives Events
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Text;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Mvc;
using Newtonsoft.Json;
namespace EmployeeRecords.Controllers
{
public class GridEvent<T> where T : class
{
public string Id { get; set; }
public string Subject { get; set; }
public string EventType { get; set; }
public T Data { get; set; }
public DateTime EventTime { get; set; }
}
[Produces("application/json")]
[Route("api/EmployeeUpdates")]
public class EmployeeUpdatesController : Controller
{
private bool EventTypeSubcriptionValidation
=> HttpContext.Request.Headers["aeg-event-type"].FirstOrDefault() ==
"SubscriptionValidation";
private bool EventTypeNotification
=> HttpContext.Request.Headers["aeg-event-type"].FirstOrDefault() ==
"Notification";
[HttpPost]
public async Task<HttpResponseMessage> Post()
{
using (var reader = new StreamReader(Request.Body, Encoding.UTF8))
{
var jsonContent = await reader.ReadToEndAsync();
var gridEvent =
JsonConvert.DeserializeObject<List<GridEvent<Dictionary<string,
string>>>>(jsonContent)
.SingleOrDefault();
if (gridEvent == null)
{
return new HttpResponseMessage { StatusCode = HttpStatusCode.BadRequest};
}
// Check the event type from Event Grid.
if (EventTypeSubcriptionValidation)
{
// Retrieve the validation code and echo back.
var validationCode = gridEvent.Data["validationCode"];
var validationResponse =
JsonConvert.SerializeObject(new { validationResponse =
validationCode });
return new HttpResponseMessage
{
StatusCode = HttpStatusCode.OK,
Content = new StringContent(validationResponse)
};
}
else if (EventTypeNotification)
{
// Pseudo code: Update records
return new HttpResponseMessage { StatusCode = HttpStatusCode.OK };
}
else
{
return new HttpResponseMessage { StatusCode = HttpStatusCode.BadRequest };
}
}
}
}
}
When creating the event subscription, the event type registered should be employeeRemoved. This change satisfies the requirement that the handler only wants to receive messages for an employee that’s removed from the organization. Also notice that neither the prefix nor suffix filters are used because the subscriber wants to be notified for each occurrence, regardless of the department:
az eventgrid event-subscription create --name <event-subscription-name> \
--resource-group <resource group name> \
--topic-name <topic name> \
--included-event-type employeeRemoved \
--endpoint <function endpoint>
Last, remember that the endpoint for the event subscription must be secure. If you reference an App Service on Azure, you have to specify HTTPS in the address, or adding the subscription will fail.
Wrapping Up
Azure Event Grid is truly a game-changing service. In this article, I addressed a common application integration scenario. Event Grid was used as the enabling technology to connect the application to other services such as an Azure Function, a Logic App and even a custom WebHook that could reside anywhere. When complemented with serverless apps, Event Grid really shines, as the two together can take advantage of the tremendous scale and integration features that Azure supports. The code in this article can be found at github.com/dbarkol/AzureEventGrid.
David Barkol is an Azure Specialist at Microsoft on the Global Black Belt Team. Contact him on Twitter: @dbarkol or through email at dabarkol@microsoft.com.
Thanks to the following Microsoft technical experts for reviewing this article: Bahram Banisadr and Dan Rosanovsanova
Dan Rosanova is the Principle Program Manager Lead in charge of the Azure Messaging suite of products including Service Bus, Event Hubs, Azure Relay, and Event Grid.
Bahram Banisadr is the PM in charge of Azure Event Grid working to build the connective tissue for Azure services.