November 2019
Volume 34 Number 11
[Data Points]
Exploring the Azure Functions Durable Entities Preview
By Julie Lerman
.jpg "Julie lerman")
Azure Functions is the Azure serverless computing offering. A function responds to some type of trigger (for example, an HttpRequest, a change in a MessageQueue or to some change in a Cosmos DB database), performs a task (which may involve collecting data from another resource) and then outputs one or more resources. The outputs can be any of the examples mentioned or others, such as sending out an SMS via a Twilio output binding. Azure Functions makes firing the triggers, reading the data and creating outputs simple by using binding configurations. A good example is interacting with Cosmos DB. With connection strings and possibly a query embedded into the configuration, a trigger binding listens to the database; an input binding reads the data for you; and an output binding knows how to insert data into the database. And there are a lot more features involved, such as built-in scalability and security. The bottom line is that Azure Functions lets you, the developer, focus on the logic of the function rather than the infrastructure around it.
Durable Azure Functions
I’ve delved into using Azure Functions in a number of previous Data Points columns. Early iterations of Azure Functions were stateless. They took in some information and pushed out some information. But developers were performing acrobatics with queues in order to combine functions into workflows. Eventually, Microsoft created a new extension for Azure Functions: Durable Functions, which allows you to orchestrate multiple functions in a workflow. Underneath the covers, the orchestration takes care of stateful information that needs to go from one function to the next, as well as any needed error handling for the workflow. There are a number of patterns that these orchestrations satisfy, such as function chaining, fan-out/fan-in, human interaction and more. “An introduction to Azure Durable Functions: Patterns and Best Practices” is an article that provides a great overview of Durable Functions and these patterns.
Coming in Version 2: Durable Entity Functions
Version 2 of Durable Functions is still in Preview (called beta2) as I’m writing this in early October 2019, though General Availability (GA) is expected soon since beta3 will be the final beta. (Don’t confuse this with Version 2 of Azure Functions, which has been available for some time.) One of the features that’s coming in this next version is a type of Azure Functions called a Durable Entity. Now, knowing my love for Entity Framework, you may think this attracted my attention because of the word “entity,” but there’s no connection between the two. Every class that has an identity is called an entity.
A durable entity, exposed by an Azure Function with a new trigger binding called EntityTrigger, has logic to manage and remember its own state. And it can be accessed and modified repeatedly by other functions.
I saw a demo of durable entities and was fascinated by the feature. They’re stateful, but you don’t have to do anything to maintain the state. By default, state is stored in Azure Storage, which is how other types of durable functions store state as well. But all this persistence is managed by the durable functions, not by you.
Currently, Durable Entities are supported only in C#, with JavaScript support planned for the GA release.
While documentation and samples will demonstrate storing and returning a single value, I got cheeky and tried to store multiple values in a durable entity and read the entire object, which also works. So you can use entity functions to persist scalars, objects, arrays, dictionaries and so forth in your serverless functions.
Before I was bold enough to be cheeky, I endured the usual bumps and bruises getting my first demos to work. Certainly, if I had just copied samples provided by the team, they’d have worked right away, but what’s the fun in that?
Preparing to Build Azure Functions
I’m going to walk you through running and debugging your first durable entity and help you avoid the pitfalls I encountered. I’ll start with a durable orchestration and then add in the entity.
You can use either Visual Studio or Visual Studio Code with the Azure Functions extension support. If you’re using Visual Studio 2017, you can install the Azure Functions and Web Jobs extension directly. In Visual Studio 2019, the extension is built into the Azure Development workload. For those using Visual Studio Code, the Azure Functions extension is what you’ll need, and be sure to follow its instructions to install the Azure Functions Core Tools. A handy tip for you: I already had the Azure Functions Core Tools installed on my machine, but it was version 2.0.3 and my functions wouldn’t start. I finally solved this by updating the tools to the latest version, which (as I’m writing) is 2.7.1633. I interacted with this demo on my Windows computer in Visual Studio 2019 and Visual Studio Code, as well as on macOS using Visual Studio Code, so you have a lot of options!
You’ll need to use AzureWebStorage, which requires an Azure account in place. Your Visual Studio subscription comes with account credits or you can create a free trial account at azure.com/free. There are emulators available, for example a Windows-only stand-alone Azure Storage Emulator from Microsoft. However, I wasn’t successful using these preview bits of entity functions with the emulator although I expect that to change within the GA time frame.
Start with a Durable Orchestration Azure Function
If you’ve never created a Durable Azure Function, I recommend starting by creating one. I’ll use Visual Studio Code and its extension. If you’re using Visual Studio and my steps don’t easily translate for you, get started here. There’s a template in both extensions that will create a function with an Orchestration class that uses the chaining pattern. Note that the template hasn’t been updated for Version 2 of Durable Functions, so you’ll be tweaking some of the generated code.
To create the new function:
- Create a folder to host the function.
- Open Visual Studio Code in that folder and use the Azure Functions extensions’s “Create new project” icon to create a function in the folder. Then follow the prompts with these steps:
- Select the desired folder where the project should be created.
- Select C# (again, for 2.0.0-beta2 of Durable Entities, only C# is supported).
- For the project type, select DurableFunctionsOrchestration.
- Specify the name of the function. I used the default.
- Specify the namespace for the function: Mine is DataPoints.Function.
- When prompted for a storage account, choose “Select storage account” and follow the prompts to select your subscription (log in if necessary) and then either create a new or select an existing storage account.
That’s it. The extension will then generate all of the needed assets for the durable function: a csproj file, a function class, a host.json class and a local.settings.json class that has the connection string for your chosen Web storage.
Understanding the Orchestration Class
The function class has three methods defined—RunOrchestrator, SayHello and HttpStart—as shown in Figure 1. You’ll find the full listing, which includes the using statements at the start of the class file, in the download sample. Each method has a function name defined with the FunctionName attribute. The RunOrchestration function (with its OrchestrationTrigger binding) calls the Hello function three times using the DurableOrchestrationContext.CallActivityAsync method. Each call adds a new string to a List<String> variable. The orchestration’s internal logic ensures that these are called in order regardless of how long each call may take to finish.
Figure 1 The New Orchestration Class with Its Three Function Methods, Before My V2 Modifications
public static class DurableFunctionsOrchestrationCSharp
{
[FunctionName("DurableFunctionsOrchestrationCSharp")]
public static async Task<List<string>> RunOrchestrator(
[OrchestrationTrigger] DurableOrchestrationContext context)
{
var outputs = new List<string>();
outputs.Add(await context.CallActivityAsync<string>(
"DurableFunctionsOrchestrationCSharp_Hello", "Tokyo"));
outputs.Add(await context.CallActivityAsync<string>(
"DurableFunctionsOrchestrationCSharp_Hello", "Seattle"));
outputs.Add(await context.CallActivityAsync<string>(
"DurableFunctionsOrchestrationCSharp_Hello", "London"));
return outputs;
}
[FunctionName("DurableFunctionsOrchestrationCSharp_Hello")]
public static string SayHello([ActivityTrigger] string name, ILogger log)
{
log.LogInformation($"Saying hello to {name}.");
return $"Hello {name}!";
}
[FunctionName("DurableFunctionsOrchestrationCSharp_HttpStart")]
public static async Task<HttpResponseMessage> HttpStart(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")]HttpRequestMessage req,
[OrchestrationClient]DurableOrchestrationClient starter,ILogger log)
{
string instanceId = await starter.StartNewAsync("DurableFunctionsOrchestrationCSharp", null);
log.LogInformation($"Started orchestration with ID = '{instanceId}'.");
return starter.CreateCheckStatusResponse(req, instanceId);
}
}
The SayHello function, which is decorated with an ActivityTrigger binding, builds the strings as requested by the orchestrator.
Orchestrations and entities can be invoked and managed using HTTP Requests, and the last function is an HttpTrigger that listens for a request. It’s called HttpStart and it also has an OrchestrationClient binding so it knows how to work with the orchestration method.
So I’ve really got three functions, not just the Durable Orchestration function. The HttpTrigger (HttpStart method) listens for an HTTP request, which then uses its OrchestrationClient (named starter) to trigger the OrchestrationContext, which then goes about its business of building up the list of strings by calling out to the SayHello function repeatedly.
Update to V2 Syntax
Since this template is for Version 1, you’ll need to make some updates for Version 2.
First, in the csproj file, change the version of the DurableTask package to 2.0.0-beta2 or whatever the latest version is. (Check NuGet for the current version.) Don’t just use 2.0.0-*. Be specific, even if this means you’ll have to update it later to any future betas or previews:
<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask"
Version="2.0.0-beta2" />
In the new class, a few types have moved and been renamed.
In the RunOrchestartor function, the DurableOrchestrationContext in the signature should become IDurableOrchestrationContext. This allows for more flexible dependency injection and testing.
For the same reasons, the DurableOrchestrationClient becomes IDurableOrchestrationClient.
Finally, change the OrchestrationClient attribute of the HttpStart method to DurableClient.
Debug the Durable Orchestration
My next recommendation is to put some breakpoints into Orchestration (line 17: var outputs, line 20: return outputs) and SayHello (line 31: log.LogInformation), then debug. When the function is ready to be hit, you’ll see a message in the output saying, “Host lock lease acquired ….” The URI of the exposed HTTP function is further up in the output. Paste that into a browser and the orchestration will get triggered.
It’s really interesting to see how durable orchestration flows. The client will cause the Orchestration function to run repeatedly, executing only one of the SayHello asynchronous calls on each loop. Using the storage, it keeps track of what has been executed. On the first loop it captures the first asynchronous call (for Tokyo), and steps through the rest, but doesn’t act on them because the first call is blocking. On the second loop, it steps through all of the code again, but this time it captures and queues up the second call (Seattle). It loops a third time and queues up the London call. Then it loops yet again, sees there are no more asynchronous function calls, and finally executes the return outputs line. If you put the breakpoint on that line, you can see that outputs is now a collection of the three strings: Hello Tokyo!, Hello Seattle! and Hello London!. The orchestrator stored the output list in AzureWebStorage, then retrieved and updated it on each pass. That’s the magic of the durable functions and it’s used in the Durable Entities feature of Durable Functions.
One last thing to note is that the browser isn’t displaying the results of the orchestrator, but, instead, the overall status of the function—provided through links output by the starter.CreateCheckStatusResponse method in the HttpFunction. When you’ve run through the function in its entirety, don’t shut it down immediately. Click the statusQueryGetUri link in the browser window. The status data will include the output from the orchestration function in a property called output. In Figure 2 you can see the final value of the outputs variable with its three strings.
Figure 2 The Results Shown by the statusQueryGetUri
{
name: "DurableFunctionsOrchestrationCSharp",
instanceId: “76366d28a19b48a48a915056c14bd458",
runtimeStatus: "Completed",
input: null,
output: [
"Hello Tokyo!",
"Hello Seattle!",
"Hello London!"
],
createdTime: "2019-10-04T20:33:49Z",
lastUpdatedTime: "2019-10-04T20:34:05Z"
}
Switch to a More Useful HTTP Response
The status is useful, but not for a real app. Let’s change the HttpFunction to return just the output.
Comment line 55 (return req.CreateCheckStatusReponse …), then below it add the code from Figure 3 to grab and update the status; check to see if the function is finished running; and then create an HttpResponseMessage that includes the output. For this demo, the message will always return a success code of OK.
Figure 3 Code to Add to Comment Line 55
DurableOrchestrationStatus status;
while (true)
{
status = await starter.GetStatusAsync(instanceId);
if (status.RuntimeStatus == OrchestrationRuntimeStatus.Completed ||
status.RuntimeStatus == OrchestrationRuntimeStatus.Failed ||
status.RuntimeStatus == OrchestrationRuntimeStatus.Terminated)
{
break;
}
}
return req.CreateResponse(System.Net.HttpStatusCode.OK, status.Output);
If you run the app again and then call the same URI as before, the browser page will now display just the output from the orchestration function, as shown in the following code:
[
"Hello Tokyo!",
"Hello Seattle!",
"Hello London!",
]
This is thanks to internal logic that waits for the orchestration to complete or timeout and then returns its outputs to the caller.
Adding a Durable Entity Function to the Mix
Durable Entities are triggered with an EntityTrigger attribute and controlled by a DurableEntityContext. You can put all of the entity’s logic inside the function if it’s super simple, but a nicer option is to create a normal class and let the function be a method in the class.
Here’s a new class that allows my apps to keep track of whatever they want to count. It’s a durable entity, thanks to the EntityTrigger function, which inherits the name of the class, Counter:
public class Counter
{
[JsonProperty("value")]
public int CurrentValue { get; set; }
public void Add(int amount) => this.CurrentValue += amount;
public void Reset() => this.CurrentValue = 0;
public int Get() => this.CurrentValue;
[FunctionName(nameof(Counter))]
public static Task Run([EntityTrigger] IDurableEntityContext ctx)
=> ctx.DispatchAsync<Counter>();
}
The counter can add to, reset or just return its CurrentValue. The CurrentValue is tucked away in AzureWebStorage and is tied to a particular instance of the counter using an ID. So you can have lots of different apps using the same function to keep track of their own counters. As mentioned earlier, Get can return something other than a scalar. I’ve even used the full class, for example, public Counter Get()=>this .
Interacting with the Durable Entity
You can call into an HttpTrigger from an orchestration or from another type of function called a durable client. As I already have the orchestration, I’ll add the counter to the existing function. The goal is to keep track of how many times that orchestration has been called. Azure’s analytics can also give you this information, but this use case suits my purposes.
In the orchestration function, add the following code between the third Hello call—the one for London—and return outputs:
var entityId = new EntityId(nameof(Counter), "HelloCounter");
context.SignalEntity(entityId, "Add", 1);
var finalValue = await context.CallEntityAsync<int>(entityId, "Get");
outputs.Add($"Hello Orchestration has been run {finalValue} times");
This defines a unique identity, “HelloCounter,” for the counter used by this orchestrator. Its data gets stored along with other information about the counter and that’s how the same durable function can have multiple entities. Earlier, I used the IDurableOrchestrationContext CallActivityAsync method to call the SayHello ActivityTrigger function. Now I’m using two other methods. The first is SignalEntity. Notice it’s not asynchronous, but rather a “fire and forget” call to the durable function. I tell it to hit the Add method for the counter tied to this particular entityId and pass in the value 1, which will increment the counter’s value. Then I call the CallEntityAsync method, which can receive a response. In its parameters, I again specify which instance of the counter and tell it to hit the Get method. Finally, I add a new string to outputs that relays the counter’s value.
That’s all the changes. Now I’ll start up the debugger again, hit the orchestrator’s function URL in the browser and let the function run its full course. The browser outputs:
[
"Hello Tokyo!",
"Hello Seattle!",
"Hello London!",
"Hello Orchestration has been run 1 times"
]
Refresh the browser and the final line will say the orchestration has run two times. Every time you refresh the browser, that number will increment.
Now stop the function completely. Then run or debug it again. Refresh the browser page and you’ll see that the number hasn’t gone back to 1; instead, it has incremented from the previous execution.
You can stop the function, close Visual Studio, even turn off the computer, and that number will be remembered and incremented each time the running function is executed. That’s durable! Better yet, if a durable function (including durable entities) is idle or the runtime scaler is zero (either by your explicit setting or from using an auto-scaling feature), you won’t be charged for compute time. The charges are only incurred when the function is running, yet the state is still retained in Azure Storage and will get picked up when the function restarts.
Calling the Counter from a Durable Client
I’ll leave you with one last trick.
Remember I said there were two ways to trigger durable entity functions? In addition to doing this from an orchestration, the other option is using a Durable Client. Let’s create a durable client to reset the orchestration’s counter.
In a new class, create this durable client function that can reset that particular counter. Notice that I’m using the same ID I use in the orchestration class:
public static class CounterReset
{
[FunctionName("CounterReset")]
public static async Task Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequest req,
[DurableClient] IDurableEntityClient client)
{
var entityId = new EntityId(nameof(Counter), "HelloCounter");
await client.SignalEntityAsync(entityId, "Reset");
}
}
Now when you run the function, it will show you two possible links to connect to: the original DurableFunctionsOrchestrationCSharp_HttpStart and the new CounterReset link. Click the CounterReset link. That function wasn’t written to return an HttpResponse after it has performed its job. However, if you browse to the orchestration’s link, you’ll see that the count is now 1, incremented from the reset value of 0.
Try Some Other Durable Functions Features
Now that you’ve gotten a taste of durable functions and the new durable entities, you should have the confidence to begin exploring more of their features. Here, I’ve only created strings in response to an HttpRequest. But remember that with Azure Functions you have the ability to leverage a variety of triggers and input and output bindings. Maybe you can have a durable entity that responds to some activity in a Cosmos DB database, or a durable orchestration that sends out SMS messages through the Twilio output binding.
Additionally, version 2.0 of Durable Functions introduces more new features such as durable HTTP and the ability to specify your durable storage mechanism.
Julie Lerman is a Microsoft Regional Director, Microsoft MVP, Docker Captain and software team coach who lives in the hills of Vermont. You can find her presenting on data access and other topics at user groups and conferences around the world. She blogs at thedatafarm.com/blog and is the author of “Programming Entity Framework,” as well as a Code First and a DbContext edition, all from O’Reilly Media. Follow her on Twitter: @julielerman and see her Pluralsight courses at bit.ly/PS-Julie.
Thanks to the following Microsoft technical expert for reviewing this article: Jeff Hollan