Durable Orchestrations

Durable Functions is an extension of Azure Functions. You can use an orchestrator function to orchestrate the execution of other Durable functions within a function app. Orchestrator functions have the following characteristics:

  • Orchestrator functions define function workflows using procedural code. No declarative schemas or designers are needed.
  • Orchestrator functions can call other durable functions synchronously and asynchronously. Output from called functions can be reliably saved to local variables.
  • Orchestrator functions are durable and reliable. Execution progress is automatically checkpointed when the function "awaits" or "yields". Local state is never lost when the process recycles or the VM reboots.
  • Orchestrator functions can be long-running. The total lifespan of an orchestration instance can be seconds, days, months, or never-ending.

This article gives you an overview of orchestrator functions and how they can help you solve various app development challenges. If you are not already familiar with the types of functions available in a Durable Functions app, read the Durable Function types article first.

Orchestration identity

Each instance of an orchestration has an instance identifier (also known as an instance ID). By default, each instance ID is an autogenerated GUID. However, instance IDs can also be any user-generated string value. Each orchestration instance ID must be unique within a task hub.

The following are some rules about instance IDs:

  • Instance IDs must be between 1 and 256 characters.
  • Instance IDs must not start with @.
  • Instance IDs must not contain /, \, #, or ? characters.
  • Instance IDs must not contain control characters.

Note

It is generally recommended to use autogenerated instance IDs whenever possible. User-generated instance IDs are intended for scenarios where there is a one-to-one mapping between an orchestration instance and some external application-specific entity, like a purchase order or a document.

An orchestration's instance ID is a required parameter for most instance management operations. They are also important for diagnostics, such as searching through orchestration tracking data in Application Insights for troubleshooting or analytics purposes. For this reason, it is recommended to save generated instance IDs to some external location (for example, a database or in application logs) where they can be easily referenced later.

Reliability

Orchestrator functions reliably maintain their execution state by using the event sourcing design pattern. Instead of directly storing the current state of an orchestration, the Durable Task Framework uses an append-only store to record the full series of actions the function orchestration takes. An append-only store has many benefits compared to "dumping" the full runtime state. Benefits include increased performance, scalability, and responsiveness. You also get eventual consistency for transactional data and full audit trails and history. The audit trails support reliable compensating actions.

Durable Functions uses event sourcing transparently. Behind the scenes, the await (C#) or yield (JavaScript) operator in an orchestrator function yields control of the orchestrator thread back to the Durable Task Framework dispatcher. The dispatcher then commits any new actions that the orchestrator function scheduled (such as calling one or more child functions or scheduling a durable timer) to storage. The transparent commit action appends to the execution history of the orchestration instance. The history is stored in a storage table. The commit action then adds messages to a queue to schedule the actual work. At this point, the orchestrator function can be unloaded from memory.

When an orchestration function is given more work to do (for example, a response message is received or a durable timer expires), the orchestrator wakes up and re-executes the entire function from the start to rebuild the local state. During the replay, if the code tries to call a function (or do any other async work), the Durable Task Framework consults the execution history of the current orchestration. If it finds that the activity function has already executed and yielded a result, it replays that function's result and the orchestrator code continues to run. Replay continues until the function code is finished or until it has scheduled new async work.

Note

In order for the replay pattern to work correctly and reliably, orchestrator function code must be deterministic. For more information about code restrictions for orchestrator functions, see the orchestrator function code constraints topic.

Note

If an orchestrator function emits log messages, the replay behavior may cause duplicate log messages to be emitted. See the Logging topic to learn more about why this behavior occures and how to work around it.

Orchestration history

The event-sourcing behavior of the Durable Task Framework is closely coupled with the orchestrator function code you write. Suppose you have an activity-chaining orchestrator function, like the following C# orchestrator function:

[FunctionName("E1_HelloSequence")]
public static async Task<List<string>> Run(
    [OrchestrationTrigger] DurableOrchestrationContext context)
{
    var outputs = new List<string>();

    outputs.Add(await context.CallActivityAsync<string>("E1_SayHello", "Tokyo"));
    outputs.Add(await context.CallActivityAsync<string>("E1_SayHello", "Seattle"));
    outputs.Add(await context.CallActivityAsync<string>("E1_SayHello", "London"));

    // returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
    return outputs;
}

If you're coding in JavaScript, your activity-chaining orchestrator function might look like the following example code:

const df = require("durable-functions");

module.exports = df.orchestrator(function*(context) {
    const output = [];
    output.push(yield context.df.callActivity("E1_SayHello", "Tokyo"));
    output.push(yield context.df.callActivity("E1_SayHello", "Seattle"));
    output.push(yield context.df.callActivity("E1_SayHello", "London"));

    // returns ["Hello Tokyo!", "Hello Seattle!", "Hello London!"]
    return output;
});

At each await (C#) or yield (JavaScript) statement, the Durable Task Framework checkpoints the execution state of the function into some durable storage backend (typically Azure Table storage). This state is what is referred to as the orchestration history.

History table

Generally speaking, the Durable Task Framework does the following at each checkpoint:

  1. Saves execution history into Azure Storage tables.
  2. Enqueues messages for functions the orchestrator wants to invoke.
  3. Enqueues messages for the orchestrator itself — for example, durable timer messages.

Once the checkpoint is complete, the orchestrator function is free to be removed from memory until there is more work for it to do.

Note

Azure Storage does not provide any transactional guarantees between saving data into table storage and queues. To handle failures, the Durable Functions storage provider uses eventual consistency patterns. These patterns ensure that no data is lost if there is a crash or loss of connectivity in the middle of a checkpoint.

Upon completion, the history of the function shown earlier looks something like the following table in Azure Table Storage (abbreviated for illustration purposes):

PartitionKey (InstanceId) EventType Timestamp Input Name Result Status
eaee885b OrchestratorStarted 2017-05-05T18:45:32.362Z
eaee885b ExecutionStarted 2017-05-05T18:45:28.852Z null E1_HelloSequence
eaee885b TaskScheduled 2017-05-05T18:45:32.670Z E1_SayHello
eaee885b OrchestratorCompleted 2017-05-05T18:45:32.670Z
eaee885b OrchestratorStarted 2017-05-05T18:45:34.232Z
eaee885b TaskCompleted 2017-05-05T18:45:34.201Z """Hello Tokyo!"""
eaee885b TaskScheduled 2017-05-05T18:45:34.435Z E1_SayHello
eaee885b OrchestratorCompleted 2017-05-05T18:45:34.435Z
eaee885b OrchestratorStarted 2017-05-05T18:45:34.857Z
eaee885b TaskCompleted 2017-05-05T18:45:34.763Z """Hello Seattle!"""
eaee885b TaskScheduled 2017-05-05T18:45:34.857Z E1_SayHello
eaee885b OrchestratorCompleted 2017-05-05T18:45:34.857Z
eaee885b OrchestratorStarted 2017-05-05T18:45:35.032Z
eaee885b TaskCompleted 2017-05-05T18:45:34.919Z """Hello London!"""
eaee885b ExecutionCompleted 2017-05-05T18:45:35.044Z "[""Hello Tokyo!"",""Hello Seattle!"",""Hello London!""]" Completed
eaee885b OrchestratorCompleted 2017-05-05T18:45:35.044Z

A few notes on the column values:

  • PartitionKey: Contains the instance ID of the orchestration.
  • EventType: Represents the type of the event. May be one of the following types:
    • OrchestrationStarted: The orchestrator function resumed from an await or is running for the first time. The Timestamp column is used to populate the deterministic value for the CurrentUtcDateTime API.
    • ExecutionStarted: The orchestrator function started executing for the first time. This event also contains the function input in the Input column.
    • TaskScheduled: An activity function was scheduled. The name of the activity function is captured in the Name column.
    • TaskCompleted: An activity function completed. The result of the function is in the Result column.
    • TimerCreated: A durable timer was created. The FireAt column contains the scheduled UTC time at which the timer expires.
    • TimerFired: A durable timer fired.
    • EventRaised: An external event was sent to the orchestration instance. The Name column captures the name of the event and the Input column captures the payload of the event.
    • OrchestratorCompleted: The orchestrator function awaited.
    • ContinueAsNew: The orchestrator function completed and restarted itself with new state. The Result column contains the value, which is used as the input in the restarted instance.
    • ExecutionCompleted: The orchestrator function ran to completion (or failed). The outputs of the function or the error details are stored in the Result column.
  • Timestamp: The UTC timestamp of the history event.
  • Name: The name of the function that was invoked.
  • Input: The JSON-formatted input of the function.
  • Result: The output of the function; that is, its return value.

Warning

While it's useful as a debugging tool, don't take any dependency on this table. It may change as the Durable Functions extension evolves.

Every time the function resumes from an await (C#) or yield (JavaScript), the Durable Task Framework reruns the orchestrator function from scratch. On each rerun, it consults the execution history to determine whether the current async operation has taken place. If the operation took place, the framework replays the output of that operation immediately and moves on to the next await (C#) or yield (JavaScript). This process continues until the entire history has been replayed. Once the current history has been replayed, the local variables will have been restored to their previous values.

Features and patterns

The next sections describe the features and patterns of orchestrator functions.

Sub-orchestrations

Orchestrator functions can call activity functions, but also other orchestrator functions. For example, you can build a larger orchestration out of a library of orchestrator functions. Or, you can run multiple instances of an orchestrator function in parallel.

For more information and for examples, see the Sub-orchestrations article.

Durable timers

Orchestrations can schedule durable timers to implement delays or to set up timeout handling on async actions. Use durable timers in orchestrator functions instead of Thread.Sleep and Task.Delay (C#) or setTimeout() and setInterval() (JavaScript).

For more information and for examples, see the Durable timers article.

External events

Orchestrator functions can wait for external events to update an orchestration instance. This Durable Functions feature often is useful for handling a human interaction or other external callbacks.

For more information and for examples, see the External events article.

Error handling

Orchestrator functions can use the error-handling features of the programming language. Existing patterns like try/catch are supported in orchestration code.

Orchestrator functions can also add retry policies to the activity or sub-orchestrator functions that they call. If an activity or sub-orchestrator function fails with an exception, the specified retry policy can automatically delay and retry the execution up to a specified number of times.

Note

If there is an unhandled exception in an orchestrator function, the orchestration instance will complete in a Failed state. An orchestration instance cannot be retried once it has failed.

For more information and for examples, see the Error handling article.

Critical sections

Orchestration instances are single-threaded so it isn't necessary to worry about race conditions within an orchestration. However, race conditions are possible when orchestrations interact with external systems. To mitigate race conditions when interacting with external systems, orchestrator functions can define critical sections using a LockAsync method in .NET.

The following sample code shows an orchestrator function that defines a critical section. It enters the critical section using the LockAsync method. This method requires passing one or more references to a Durable Entity, which durably manages the lock state. Only a single instance of this orchestration can execute the code in the critical section at a time.

[FunctionName("Synchronize")]
public static async Task Synchronize(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    var lockId = new EntityId("LockEntity", "MyLockIdentifier");
    using (await context.LockAsync(lockId))
    {
        // critical section - only one orchestration can enter at a time
    }
}

The LockAsync acquires the durable lock(s) and returns an IDisposable that ends the critical section when disposed. This IDisposable result can be used together with a using block to get a syntactic representation of the critical section. When an orchestrator function enters a critical section, only one instance can execute that block of code. Any other instances that try to enter the critical section will be blocked until the previous instance exits the critical section.

The critical section feature is also useful for coordinating changes to durable entities. For more information about critical sections, see the Durable entities "Entity coordination" topic.

Note

Critical sections are available in Durable Functions 2.0 and above. Currently, only .NET orchestrations implement this feature.

Calling HTTP endpoints

Orchestrator functions aren't permitted to do I/O, as described in orchestrator function code constraints. The typical workaround for this limitation is to wrap any code that needs to do I/O in an activity function. Orchestrations that interact with external systems frequently use activity functions to make HTTP calls and return the result to the orchestration.

To simplify this common pattern, orchestrator functions can use the CallHttpAsync method in .NET to invoke HTTP APIs directly. In addition to supporting basic request/response patterns, CallHttpAsync supports automatic handling of common async HTTP 202 polling patterns, and also supports authentication with external services using Managed Identities.

[FunctionName("CheckSiteAvailable")]
public static async Task CheckSiteAvailable(
    [OrchestrationTrigger] IDurableOrchestrationContext context)
{
    Uri url = context.GetInput<Uri>();

    // Makes an HTTP GET request to the specified endpoint
    DurableHttpResponse response = 
        await context.CallHttpAsync(HttpMethod.Get, url);

    if (response.StatusCode >= 400)
    {
        // handling of error codes goes here
    }
}

For more information and for detailed examples, see the HTTP features article.

Note

Calling HTTP endpoints directly from orchestrator functions is available in Durable Functions 2.0 and above. Currently, only .NET orchestrations implement this feature.

Passing multiple parameters

It isn't possible to pass multiple parameters to an activity function directly. The recommendation is to pass in an array of objects or to use ValueTuples objects in .NET.

The following sample is using new features of ValueTuples added with C# 7:

[FunctionName("GetCourseRecommendations")]
public static async Task<object> RunOrchestrator(
    [OrchestrationTrigger] DurableOrchestrationContext context)
{
    string major = "ComputerScience";
    int universityYear = context.GetInput<int>();

    object courseRecommendations = await context.CallActivityAsync<object>(
        "CourseRecommendations",
        (major, universityYear));
    return courseRecommendations;
}

[FunctionName("CourseRecommendations")]
public static async Task<object> Mapper([ActivityTrigger] DurableActivityContext inputs)
{
    // parse input for student's major and year in university
    (string Major, int UniversityYear) studentInfo = inputs.GetInput<(string, int)>();

    // retrieve and return course recommendations by major and university year
    return new
    {
        major = studentInfo.Major,
        universityYear = studentInfo.UniversityYear,
        recommendedCourses = new []
        {
            "Introduction to .NET Programming",
            "Introduction to Linux",
            "Becoming an Entrepreneur"
        }
    };
}

Next steps