Run containers in a hybrid environment

Container Registry

Arc

Container Instances

Cosmos DB

Kubernetes Service

This reference architecture illustrates how developers can create, manage, and monitor deployed containers in the public cloud, across multiple clouds, and on-premises.

Architecture

The diagram illustrates a developer team that deploys its container images to a Microsoft Azure Container Registry. Subsequently, the container images are pulled and deployed to either an on-premises or cloud-based Kubernetes cluster. The containers are monitored using Azure Monitor and the container images are scanned and monitored using Azure Container Registry.

Download a Visio file of this architecture.

Workflow

Microsoft Azure Container Registry (ACR). ACR is a service that creates a managed registry. ACR builds, stores, and manages container images and can store containerized machine learning models.
Azure Kubernetes Service (AKS). AKS is a managed service that offers a managed Kubernetes cluster with elastic scale-out functionality.
Azure Container Instances. Azure Container Instances runs containers on-demand in a serverless Microsoft Azure environment. Azure Container Instances is a low-friction method of running containers that doesn't require a full Docker host or Kubernetes installation.
Azure Cosmos DB. Azure Cosmos DB is a multiple model database that can serve data elastically at a massive scale. Azure Cosmos DB was designed for applications that are globally distributed in a multi-write model.
Azure Key Vault. Azure Key Vault is a hardware-backed credential management service that has tight integration with Microsoft identity services and compute resources.
Azure Policy. Azure Policy enforces standards and assesses compliance for targeted resources deployed to Azure.
Azure Private Link. Azure Private Link creates a private endpoint in your virtual network that you can use to communicate with Azure platform as a service (PaaS) without exposing your service to the public internet.
Azure Monitor. Azure Monitor is an all-encompassing suite of monitoring services for applications that deploy both in Azure and on-premises.
Microsoft Defender for Cloud. Microsoft Defender for Cloud is a unified security management and threat protection system for workloads across on-premises, multiple clouds, and Azure.
On-premises Kubernetes cluster. In this architecture, a local Kubernetes cluster is used to run multiple containers on-premises.

Scenario details

Typical uses for this architecture include:

Web applications with internal and external components that deploy both to the public cloud and on-premises by using shared container images.
Modern deployment testing cycles with quality analysis, testing, development, or staging that's hosted on-premises and in the public cloud.

Recommendations

Azure Container Registry

ACR is an enterprise container registry that can implement common best practices by protecting images from unauthorized access, replicating images across multiple geographies, preventing unnecessary ingress/egress, and optimizing costs. Additionally, ACR supports turnkey geo-replication across multiple Azure regions, which helps you minimize latency between ACR, your container hosts, and your development team.

ACR includes a suite of tasks, referred to as ACR Tasks, that can manage cloud-based container image building and maintenance across a variety of operating systems. ACR Tasks can be triggered manually, by a change to source control, by a change to the base container image, or on a fixed schedule. The following are scenarios in which you could use ACR Tasks:

An Internet of Things (IoT) developer is building container images to run on ARM-based IoT devices. The developer might be using a Linux or macOS operating system to develop the software, but they will need to perform the build on an ARM platform.
A software as a service (SaaS) development team is building software on Windows computers that will run their container images on Linux hosts. The team would like its build to be performed on a Linux host.
An open source project maintainer is building a container image that augments a well-known operating system base image. The maintainer will likely want their container image to update every time the base image updates.

Note

ACR Tasks can standardize the build environment and perform continuous integration of your container images.

Azure Container Instances

Azure Container Instances is a low-friction, serverless compute environment for containerized applications. Azure Container Instances is an excellent choice for container deployment because of its low management overhead and quick startup times. Container images that are stored in ACR can deploy directly to Azure Container Instances container groups.

In this reference, Azure Container Instances container groups are utilized as virtual nodes for an Azure Kubernetes Service cluster. AKS uses virtual nodes to register a virtual pod with unlimited capacity and the ability to dispatch pods by using Azure Container Instances container groups. This is ideal for scenarios where you want very fast provisioning of individual pods and only want to pay for the execution time per second.

Considerations

Scalability

Customer-facing containerized web applications benefit from variable scales. You can use services such as Azure Container Instances and AKS to dynamically scale out to meet anticipated or measured demand. Additionally, you can use services such as Azure Functions and Azure App Service to run container images at scale.
Internal application usage is more predictable and can run on an existing Kubernetes cluster. If you're interested in deploying Azure-managed services on-premises, consider:
- Connecting a Kubernetes cluster to Azure Arc.
- Deploying Functions or App Service in Azure Stack Hub.
Azure Cosmos DB automatically scales service resources to meet the storage needs of your application in an elastic manner. For throughput, you can choose to pre-provision throughput or operate Azure Cosmos DB as a serverless service. If your workload has variable or unpredictable demands, you can also choose to provision your throughput using autoscale.

Availability

Modern applications typically include a website, one or more HTTP APIs, and some connection to a data store. Applications within a container image should be designed to be stateless for maximum horizontal scale and availability. Any data should be stored in a separate service that has similar availability. For guidance on designing an application that can scale to thousands of nodes, refer to the performance efficiency section of the Azure Well-Architected Framework.
AKS has a reference architecture baseline that defines each of the Well-Architected Framework categories and recommends an implementation in line with the category.
To reduce the impact of large pulls of container images, deploy ACR in a region that's closest to the development team and the production compute services. Consider a geo-replicated ACR deployment for distributed teams and distributed production containers.
Azure Cosmos DB is a database service that supports turnkey global distribution and supports automatic failover across multiple regions. Azure Cosmos DB also has the ability to enable multiple region writes and dynamically add or remove regions.

Manageability

Consider using Azure Resource Manager templates to deploy Azure Container Instance container groups in a repeatable fashion for multiple region deployments and large-scale orchestration. You can similarly use Azure Resource Manager templates to deploy Azure Kubernetes Service, Azure Key Vault, and Azure Cosmos DB.
Consider utilizing Azure role-based access control (Azure RBAC) to prevent users from accidentally creating or deleting container instances without permission.
Use Azure Monitor to monitor metrics and logs for both on-premises and remote containers, analyze the data using queries, and create alerts for abnormal situations.
Use Azure Policy to implement enforcement of a set of rules for clusters and pods deployed to Kubernetes Service or an Azure Arc-enabled Kubernetes cluster.

DevOps

Use ACR Tasks to automate the build of container images on a schedule or when changes are made to the source code. Additionally, consider using ACR Tasks to automatically update container images as base images are patched and updated.
The AKS team has developed GitHub actions that can assist with implementing GitOps and facilitate deployments from ACR to AKS clusters.
If your Kubernetes cluster is attached to Azure Arc, you can manage your Kubernetes cluster using GitOps. To review best practices for connecting a hybrid Kubernetes cluster to Azure Arc, refer to the Azure Arc hybrid management and deployment for Kubernetes clusters reference architecture.

Security

Use Azure Private Link to communicate to and across services in your virtual network. This will route traffic through specific subnets to reach the individual Azure services directly and protect your data from inadvertent exposure to the public internet.

Cost optimization

Use the Azure pricing calculator to estimate costs.
If your development team and production instances are in a single region, consider placing the Container Registry resource in the same region. This will allow you to minimize container push and pull latency and avoid the additional costs associated with the Premium Azure Container Registry service tier.
Configuring ACR to use an Azure Virtual Network through an Azure Private Link service endpoint requires the ACR instance to be deployed in the Premium tier.
AKS offers free cluster management. Billing is isolated to the compute, storage, and networking resources used by AKS to host nodes. Refer to Azure Virtual Machine or Azure Container Instances pricing to review specific pricing details for each compute service.
If you require a specific uptime service-level agreement (SLA), you can enable the uptime SLA optional feature of AKS.
Azure Container Instances resources are billed per second, based on an allocation of virtual CPU and memory resources, to the container group. Allocating unnecessary compute resources can exponentially increase the costs required to run this architecture solution. Cost monitoring and optimization is a continuous process that should be conducted at regular intervals throughout the lifetime of your deployment. For further guidance on minimizing Azure Container Instances operational costs, refer to the cost optimization section of the Azure Well-Architected Framework.