Serverless batch processing using Durable Functions with Azure Container Instances

This article describes how you can use Azure Functions Durable Functions to schedule, manage, and deploy serverless batch processing jobs in Azure Container Instances (ACI) containers.

Containers are popular for packaging, deploying, and managing microservices-based architectures. Orchestration is the task of automating and managing containers and their interactions. Popular container orchestrator platforms like Azure Kubernetes Service (AKS) and Azure Service Fabric can manage complex, multi-container tasks and interactions.

You don't always need full-fledged container orchestrators to provision and manage simple apps and jobs in isolated containers. ACI is the fastest and simplest way to run containers in Azure, and Durable Functions can orchestrate container deployment.

Use cases

  • Simple workloads like batch processing jobs that use only one container image.
  • Workloads where computing needs vary depending on each individual job. For example, batch processing job #1 may need 100 container instances with large CPU and memory requirements, and take days to run, whereas job #2 needs only a couple of instances with less CPU and memory, and takes only hours to run.
  • Multi-tenant scenarios where some tenants need large computing power, while other tenants have small computing requirements.

Architecture

Durable Functions orchestration of Container Instances

Download a Visio file of this architecture.

  1. The batch processing job is packaged into a container image stored in Azure Container Registry (ACR), ready to deploy with options like AKS, Service Fabric, or ACI.
  2. An HTTP trigger invokes the orchestrator function to orchestrate the container deployment.
  3. An activity function uses the container image stored in ACR to create an ACI container in a container group.
  4. The orchestrator function uses the container URL to call and start the batch processing job on the container instance, and to monitor job progress.
  5. Once the job completes, the batch processing job invokes the orchestrator function by raising an external event, and provides job status Completed or Failed.
  6. Depending on job status, the orchestrator function stops, restarts, or deletes the container group.

Components

Alternatives

  • You can also control the container instances by using restart policies.

  • To manage complex, multi-container tasks and interactions, consider using a full-fledged container orchestrator like AKS or Azure Service Fabric.

Considerations

  • For long-term, stable workloads, orchestrating containers in a cluster of dedicated virtual machines (VMs) is usually less expensive than using ACI. However, ACI can quickly expand and contract your overall capacity to meet temporary or unexpected usage requirements. Because they start quickly and bill by the second, container instances can be the fastest and most cost effective way to get started and to handle highly variable workloads.

  • With this solution, rather than scaling out the number of VMs and then deploying more containers onto those VMs, you simply deploy additional containers as needed, and delete them when they're no longer needed. You use Durable Functions to schedule and manage the container deployment and deletion.

  • ACI enables a layered approach to orchestration by providing all of the scheduling and management to run single containers, allowing orchestrator platforms to manage multi-container tasks and architectures like scaling and coordinated upgrades.

Implementation

The .NET Core C# sample application uses Azure Durable Functions to orchestrate ACI container deployment, monitoring, and cleanup.

Next steps