GitOps Flux v2 configurations with AKS and Azure Arc-enabled Kubernetes
Azure provides configuration management capability using GitOps in Azure Kubernetes Service (AKS) and Azure Arc-enabled Kubernetes clusters. You can easily enable and use GitOps in these clusters.
With GitOps, you declare the desired state of your Kubernetes clusters in files in Git repositories. The Git repositories may contain the following files:
- YAML-formatted manifests that describe Kubernetes resources (such as Namespaces, Secrets, Deployments, and others)
- Helm charts for deploying applications
- Kustomize files to describe environment-specific changes
Because these files are stored in a Git repository, they're versioned, and changes between versions are easily tracked. Kubernetes controllers run in the clusters and continually reconcile the cluster state with the desired state declared in the Git repository. These operators pull the files from the Git repositories and apply the desired state to the clusters. The operators also continuously assure that the cluster remains in the desired state.
GitOps on Azure Arc-enabled Kubernetes or Azure Kubernetes Service uses Flux, a popular open-source tool set. Flux provides support for common file sources (Git and Helm repositories, Buckets) and template types (YAML, Helm, and Kustomize). Flux also supports multi-tenancy and deployment dependency management, among other features.
Flux cluster extension
GitOps is enabled in an Azure Arc-enabled Kubernetes or AKS cluster as a
Microsoft.KubernetesConfiguration/extensions/microsoft.flux cluster extension resource. The
microsoft.flux extension must be installed in the cluster before one or more
fluxConfigurations can be created. The extension will be installed automatically when you create the first
Microsoft.KubernetesConfiguration/fluxConfigurations in a cluster, or you can install it manually using the portal, the Azure CLI (az k8s-extension create --extensionType=microsoft.flux), ARM template, or REST API.
microsoft.flux extension installs by default the Flux controllers (Source, Kustomize, Helm, Notification) and the FluxConfig CRD, fluxconfig-agent, and fluxconfig-controller. You can control which of these controllers is installed and can optionally install the Flux image-automation and image-reflector controllers, which provide functionality around updating and retrieving Docker images.
- Flux Source controller: Watches the source.toolkit.fluxcd.io custom resources. Handles the synchronization between the Git repositories, Helm repositories, and Buckets. Handles authorization with the source for private Git and Helm repos. Surfaces the latest changes to the source through a tar archive file.
- Flux Kustomize controller: Watches the
kustomization.toolkit.fluxcd.iocustom resources. Applies Kustomize or raw YAML files from the source onto the cluster.
- Flux Helm controller: Watches the
helm.toolkit.fluxcd.iocustom resources. Retrieves the associated chart from the Helm Repository source surfaced by the Source controller. Creates the
HelmChartcustom resource and applies the
HelmReleasewith given version, name, and customer-defined values to the cluster.
- Flux Notification controller: Watches the
notification.toolkit.fluxcd.iocustom resources. Receives notifications from all Flux controllers. Pushes notifications to user-defined webhook endpoints.
- Flux Custom Resource Definitions:
- FluxConfig CRD: Custom Resource Definition for
fluxconfigs.clusterconfig.azure.comcustom resources that define
- fluxconfig-agent: Responsible for watching Azure for new or updated
fluxConfigurationsresources, and for starting the associated Flux configuration in the cluster. Also, is responsible for pushing Flux status changes in the cluster back to Azure for each
- fluxconfig-controller: Watches the
fluxconfigs.clusterconfig.azure.comcustom resources and responds to changes with new or updated configuration of GitOps machinery in the cluster.
microsoft.flux extension is installed in the
flux-system namespace and has cluster-wide scope. The option to install this extension at the namespace scope is not available, and attempt to install at namespace scope will fail with 400 error.
You create Flux configuration resources (
Microsoft.KubernetesConfiguration/fluxConfigurations) to enable GitOps management of the cluster from your Git repos or Bucket sources. When you create a
fluxConfigurations resource, the values you supply for the parameters, such as the target Git repo, are used to create and configure the Kubernetes objects that enable the GitOps process in that cluster. To ensure data security, the
fluxConfigurations resource data is stored encrypted at rest in an Azure Cosmos DB database by the Cluster Configuration service.
fluxconfig-controller agents, installed with the
microsoft.flux extension, manage the GitOps configuration process.
fluxconfig-agent is responsible for:
- Polls the Kubernetes Configuration data plane service for new or updated
- Creates or updates
FluxConfigcustom resources in the cluster with the configuration information.
FluxConfigcustom resources and pushes status changes back to the associated Azure fluxConfiguration resources.
fluxconfig-controller is responsible for:
- Watches status updates to the Flux custom resources created by the managed
- Creates private/public key pair that exists for the lifetime of the
fluxConfigurations. This key is used for authentication if the URL is SSH based and if the user doesn't provide their own private key during creation of the configuration.
- Creates custom authentication secret based on user-provided private-key/http basic-auth/known-hosts/no-auth data.
- Sets up RBAC (service account provisioned, role binding created/assigned, role created/assigned).
Bucketcustom resource and
Kustomizationcustom resources from the information in the
fluxConfigurations resource in Azure will be associated in a Kubernetes cluster with one Flux
Bucket custom resource and one or more
Kustomization custom resources. When you create a
fluxConfigurations resource, you'll specify, among other information, the URL to the source (Git repository or Bucket) and the sync target in the source for each
Kustomization. You can configure dependencies between
Kustomization custom resources to control deployment sequencing. Also, you can create multiple namespace-scoped
fluxConfigurations resources on the same cluster for different applications and app teams.
fluxconfig-agentmonitors for new or updated
fluxConfigurationresources in Azure. The agent requires connectivity to Azure for the desired state of the
fluxConfigurationto be applied to the cluster. If the agent is unable to connect to Azure, there will be a delay in making the changes in the cluster until the agent can connect. If the cluster is disconnected from Azure for more than 48 hours, then the request to the cluster will time-out, and the changes will need to be re-applied in Azure.
- Sensitive customer inputs like private key and token/password are stored for less than 48 hours in the Kubernetes Configuration service. If you update any of these values in Azure, assure that your clusters connect with Azure within 48 hours.
GitOps with Private Link
If you've added support for private link to an Azure Arc-enabled Kubernetes cluster, then the
microsoft.flux extension works out-of-the-box with communication back to Azure. For connections to your Git repository, Helm repository, or any other endpoints that are needed to deploy your Kubernetes manifests, you will need to provision these endpoints behind your firewall or list them on your firewall so that the Flux Source controller can successfully reach them.
For more information on private link scopes in Azure Arc, refer to this document.
The Azure GitOps service (Azure Kubernetes Configuration Management) stores/processes customer data. By default, customer data is replicated to the paired region. For the regions Singapore, East Asia, and Brazil South, all customer data is stored and processed in the region.
Advance to the next tutorial to learn how to enable GitOps on your AKS or Azure Arc-enabled Kubernetes clusters
Submit and view feedback for