Run containers in a hybrid environment

Container Registry
Arc
Container Instances
Cosmos DB
Kubernetes Service

This reference architecture illustrates how developers can create, manage, and monitor deployed containers in the public cloud, across multiple clouds, and on-premises.

Architecture

The diagram illustrates a developer team that deploys its container images to a Microsoft Azure Container Registry. Subsequently, the container images are pulled and deployed to either an on-premises or cloud-based Kubernetes cluster. The containers are monitored using Azure Monitor and the container images are scanned and monitored using Azure Container Registry.

Download a Visio file of this architecture.

Workflow

  • Microsoft Azure Container Registry (ACR). ACR is a service that creates a managed registry. ACR builds, stores, and manages container images and can store containerized machine learning models.
  • Azure Kubernetes Service (AKS). AKS is a managed service that offers a managed Kubernetes cluster with elastic scale-out functionality.
  • Azure Container Instances. Azure Container Instances runs containers on-demand in a serverless Microsoft Azure environment. Azure Container Instances is a low-friction method of running containers that doesn't require a full Docker host or Kubernetes installation.
  • Azure Cosmos DB. Azure Cosmos DB is a multiple model database that can serve data elastically at a massive scale. Azure Cosmos DB was designed for applications that are globally distributed in a multi-write model.
  • Azure Key Vault. Azure Key Vault is a hardware-backed credential management service that has tight integration with Microsoft identity services and compute resources.
  • Azure Policy. Azure Policy enforces standards and assesses compliance for targeted resources deployed to Azure.
  • Azure Private Link. Azure Private Link creates a private endpoint in your virtual network that you can use to communicate with Azure platform as a service (PaaS) without exposing your service to the public internet.
  • Azure Monitor. Azure Monitor is an all-encompassing suite of monitoring services for applications that deploy both in Azure and on-premises.
  • Microsoft Defender for Cloud. Microsoft Defender for Cloud is a unified security management and threat protection system for workloads across on-premises, multiple clouds, and Azure.
  • On-premises Kubernetes cluster. In this architecture, a local Kubernetes cluster is used to run multiple containers on-premises.

Scenario details

Typical uses for this architecture include:

  • Web applications with internal and external components that deploy both to the public cloud and on-premises by using shared container images.
  • Modern deployment testing cycles with quality analysis, testing, development, or staging that's hosted on-premises and in the public cloud.

Recommendations

Azure Container Registry

ACR is an enterprise container registry that can implement common best practices by protecting images from unauthorized access, replicating images across multiple geographies, preventing unnecessary ingress/egress, and optimizing costs. Additionally, ACR supports turnkey geo-replication across multiple Azure regions, which helps you minimize latency between ACR, your container hosts, and your development team.

ACR includes a suite of tasks, referred to as ACR Tasks, that can manage cloud-based container image building and maintenance across a variety of operating systems. ACR Tasks can be triggered manually, by a change to source control, by a change to the base container image, or on a fixed schedule. The following are scenarios in which you could use ACR Tasks:

  • An Internet of Things (IoT) developer is building container images to run on ARM-based IoT devices. The developer might be using a Linux or macOS operating system to develop the software, but they will need to perform the build on an ARM platform.
  • A software as a service (SaaS) development team is building software on Windows computers that will run their container images on Linux hosts. The team would like its build to be performed on a Linux host.
  • An open source project maintainer is building a container image that augments a well-known operating system base image. The maintainer will likely want their container image to update every time the base image updates.

Note

ACR Tasks can standardize the build environment and perform continuous integration of your container images.

Azure Container Instances

Azure Container Instances is a low-friction, serverless compute environment for containerized applications. Azure Container Instances is an excellent choice for container deployment because of its low management overhead and quick startup times. Container images that are stored in ACR can deploy directly to Azure Container Instances container groups.

In this reference, Azure Container Instances container groups are utilized as virtual nodes for an Azure Kubernetes Service cluster. AKS uses virtual nodes to register a virtual pod with unlimited capacity and the ability to dispatch pods by using Azure Container Instances container groups. This is ideal for scenarios where you want very fast provisioning of individual pods and only want to pay for the execution time per second.

Considerations

Scalability

Availability

  • Modern applications typically include a website, one or more HTTP APIs, and some connection to a data store. Applications within a container image should be designed to be stateless for maximum horizontal scale and availability. Any data should be stored in a separate service that has similar availability. For guidance on designing an application that can scale to thousands of nodes, refer to the performance efficiency section of the Azure Well-Architected Framework.
  • AKS has a reference architecture baseline that defines each of the Well-Architected Framework categories and recommends an implementation in line with the category.
  • To reduce the impact of large pulls of container images, deploy ACR in a region that's closest to the development team and the production compute services. Consider a geo-replicated ACR deployment for distributed teams and distributed production containers.
  • Azure Cosmos DB is a database service that supports turnkey global distribution and supports automatic failover across multiple regions. Azure Cosmos DB also has the ability to enable multiple region writes and dynamically add or remove regions.

Manageability

DevOps

Security

  • Use Azure Private Link to communicate to and across services in your virtual network. This will route traffic through specific subnets to reach the individual Azure services directly and protect your data from inadvertent exposure to the public internet.

Cost optimization

  • Use the Azure pricing calculator to estimate costs.
  • If your development team and production instances are in a single region, consider placing the Container Registry resource in the same region. This will allow you to minimize container push and pull latency and avoid the additional costs associated with the Premium Azure Container Registry service tier.
  • Configuring ACR to use an Azure Virtual Network through an Azure Private Link service endpoint requires the ACR instance to be deployed in the Premium tier.
  • AKS offers free cluster management. Billing is isolated to the compute, storage, and networking resources used by AKS to host nodes. Refer to Azure Virtual Machine or Azure Container Instances pricing to review specific pricing details for each compute service.
  • If you require a specific uptime service-level agreement (SLA), you can enable the uptime SLA optional feature of AKS.
  • Azure Container Instances resources are billed per second, based on an allocation of virtual CPU and memory resources, to the container group. Allocating unnecessary compute resources can exponentially increase the costs required to run this architecture solution. Cost monitoring and optimization is a continuous process that should be conducted at regular intervals throughout the lifetime of your deployment. For further guidance on minimizing Azure Container Instances operational costs, refer to the cost optimization section of the Azure Well-Architected Framework.

Next steps