Build, test, and push Docker container apps

Azure Pipelines | Azure DevOps Server 2019 | TFS 2018 | TFS 2017

This guidance explains how to automatically build Docker images and push them to registries such as Docker Hub or Azure Container Registry.

Note

In Microsoft Team Foundation Server (TFS) 2018 and previous versions, build and release pipelines are called definitions, service connections are called service endpoints, stages are called environments, and jobs are called phases.

Note

This guidance applies to TFS version 2017.3 and newer.

Note

This article helps you start using Azure Pipelines by using Docker commands. As an alternative, Azure Pipelines has a built-in Docker task that you can use to build and push the container images to a container registry. Learn more about how the task helps with Docker best practices and standards.

Get started

Follow these instructions to set up a pipeline for a sample Docker application.

Get the code

If you already have an app in GitHub that you want to deploy, you can try creating a pipeline for that code.

However, if you are a new user, then you might get a better start by using our sample code. In that case, fork this repo in GitHub:

https://github.com/MicrosoftDocs/pipelines-dotnet-core-docker

Create an Azure Container Registry

Create an Azure container registry if you do not have one already.

Sign in to Azure Pipelines

Sign in to Azure Pipelines. After you sign in, your browser goes to https://dev.azure.com/my-organization-name and displays your Azure DevOps dashboard.

Within your selected organization, create a project. If you don't have any projects in your organization, you see a Create a project to get started screen. Otherwise, select the Create Project button in the upper-right corner of the dashboard.

Create a pipeline

  1. Sign in to your Azure DevOps organization and navigate to your project.

  2. Go to Pipelines, and then select New Pipeline.

  3. Walk through the steps of the wizard by first selecting GitHub as the location of your source code.

    Select GitHub

    Note

    If this is not what you see, then make sure the Multi-stage pipelines experience is turned on.

  4. You might be redirected to GitHub to sign in. If so, enter your GitHub credentials.

  5. When the list of repositories appears, select your repository.

  6. You might be redirected to GitHub to install the Azure Pipelines app. If so, select Approve and install.

When the Configure tab appears, select Docker. The Docker template requires additional inputs.

  1. Select the Azure subscription to push the image to. Then, select the Azure container registry that you created above.

  2. Select Validate and configure to generate the YAML file.

  3. Take a look at the pipeline YAML file to see what it does.

  4. After you've looked at what the pipeline does, select Save and run to see the pipeline in action.

  5. Select Save and run, after which you're prompted for a commit message because Azure Pipelines adds the azure-pipelines.yml file to your repository. After editing the message, select Save and run again.

You just created and ran a pipeline that we automatically created for you, because your code appeared to be a good match for the Docker template.

When you're done, you'll have a working YAML file (azure-pipelines.yml) in your repository that's ready for you to customize.

Tip

To make changes to the YAML file as described in this topic, select the pipeline in the Pipelines page, and then Edit the azure-pipelines.yml file.

  1. The code in the following repository is a project with a Dockerfile that you can use to follow along. Fork this repo into your GitHub account.

    https://github.com/MicrosoftDocs/pipelines-dotnet-core-docker
    
  2. After you have the sample code in your own repository, create a pipeline by using the instructions in Create your first pipeline and select the Empty template. (Do not select the Docker template.)

  3. Add a Bash task at the end of the pipeline and configure it as follows to build and publish an image by using the Dockerfile in the repository:

    • Type: Inline
    • Script:

    To push to Docker Hub, use the following script:

    docker build -t $(dockerId)/$(imageName) .
    docker login -u $(dockerId) -p $(dockerPassword)
    docker push $(dockerId)/$(imageName)
    

    To push to Azure Container Registry, use the following script:

    docker build -t $(dockerId).azurecr.io/$(imageName) .
    docker login -u $(dockerId) -p $(dockerPassword) $(dockerId).azurecr.io
    docker push $(dockerId).azurecr.io/$(imageName)
    
  4. In the Variables tab of the build pipeline, define two variables:

    • imageName: $(Build.DefinitionName).$(Build.BuildId)
    • dockerId: Your Docker ID.
    • dockerPassword: Your Docker password. Mark this variable as a secret variable.

    If you use Azure Container Registry, make sure that you have pre-created the registry in the Azure portal. You can get the user ID and password from the Access keys section of the registry in the Azure portal.

  5. Save the pipeline and queue a build to see it in action.

Read through the rest of this topic to learn some of the common ways to customize your Docker pipeline.

Build environment

You can use Azure Pipelines to build and push your Docker images without needing to set up any infrastructure of your own. You can build either Windows or Linux container images. The Microsoft-hosted agents in Azure Pipelines have Docker pre-installed on them. We frequently update the version of Docker on these agent machines. To know which version of Docker is installed, see Microsoft-hosted agents.

Update the following snippet in your azure-pipelines.yml file to select the appropriate image.

pool:
  vmImage: 'ubuntu-16.04' # other options: 'macOS-10.13', 'vs2017-win2016'

See Microsoft-hosted agents for a complete list of images.

Microsoft-hosted Linux agents

When you use the Microsoft-hosted Linux agents, you get a fresh Linux virtual machine with each build. This virtual machine runs the agent and acts as a Docker host.

Microsoft-hosted VS2017 (Windows) agents

Use the Windows agents (win1803) to build Windows container images. When you use this pool, you get a fresh Windows Server 2016 virtual machine with each build. The virtual machine runs the agent and acts as a Docker host. Some of the common images, such as microsoft/dotnet-framework, microsoft/aspnet, microsoft/aspnetcore-build, and microsoft/windowsservercore, are pre-cached on this Docker host. Building new images from these images will therefore be faster. For a complete list of pre-cached images, see Azure Pipelines Windows Container 1803 image.

Microsoft-hosted MacOS agents

You cannot use macOS agents to build container images because Docker is not installed on these agents.

Self-hosted agents

As an alternative to using Microsoft-hosted agents, you can set up self-hosted agents with Docker installed. This is useful if you want to cache additional images on the Docker host and further improve the performance of your builds.

Your builds run on a self-hosted agent. Make sure that you have Docker installed on the agent.

Build an image

You can build a Docker image by running the docker build command in a script or by using the Docker task.

To run the command in a script, add the following snippet to your azure-pipelines.yml file.

steps:
- script: docker build -t $(dockerId)/$(imageName) .  # add options to this command to meet your needs

You can run any docker commands as part of the script block in your YAML file. If your Dockerfile depends on another image from a protected registry, you have to first run a docker login command in your script. If you want to avoid managing the username and password as a secret, you can use the Docker task, which uses the service connection for docker login. After you have used the Docker task to log in, the session is maintained for the duration of the job. You can then use follow-up tasks to execute any scripts.

steps:
- script: docker login -u $(dockerId) -p $(pswd) <docker-registry-url>

Make sure that you define the Docker password as a secret variable in the build pipeline and not in the YAML file.

You don't need to specify docker-registry-url in the login command, if you are connecting to Docker Hub.

  1. Select Tasks in your build pipeline, and then add the Docker task to the job.

    Note

    The Docker task supports:

    • Docker best practices: By writing minimal YAML, you can build and push an image that's tagged and labeled with rich metadata (for example, build or commit).
    • Docker standards: Work with a private registry like Azure Container Registry easily by tagging and naming an image with the registry host name and port. The task helps you to follow Docker naming conventions, for example, converting uppercase characters to lowercase and removing spaces in the image name.
    • Secret management: The task makes it easy to use a service connection for connecting to any private container registry. For example, in the case of Azure Container Registry, this helps you avoid enabling the admin user and subsequently managing the username and password as a secret. After you have used the Docker task to log in, the session is maintained for the duration of the job. You can then use follow-up tasks to run any scripts that use the login done by the Docker task.
  2. Select the task, and then for Action, select Build an image.

  3. Specify the connection to the registry that you plan to push the image to by selecting the Container registry type - Container Registry or Azure Container Registry. Then enter the properties for that connection type. If you plan to push the image to an Azure container registry, make sure that you pre-create the registry in the Azure portal.

Integrate build and test tasks

Often you'll want to build and test your app before creating the Docker image. You can orchestrate this process either in your build pipeline or in your Dockerfile.

Build and test in your build pipeline

In this approach, you use the build pipeline to orchestrate building your code, running your tests, and creating an image. This approach is useful if you want to:

  • Use tasks (either built-in tasks or those you get from the Azure DevOps Marketplace) to define the pipeline used to build and test your app.
  • Run tasks that require authentication via service connections (for example: authenticated NuGet or npm feeds).
  • Publish test results. The test results published in the example can be viewed under the Tests tab in the build.

To create an image, you run a docker build command at the end of your build pipeline. Your Dockerfile contains the instructions to copy the results of your build into the container.

Build and test in your Dockerfile

In this approach, you use your Dockerfile to build your code and run tests. The build pipeline has a single step to run docker build. The rest of the steps are orchestrated by the Docker build process. It's common to use a multi-stage Docker build in this approach. The advantage of this approach is that your build process is entirely configured in your Dockerfile. This means your build process is portable from the development machine to any build system. One disadvantage is that you can't use Azure Pipelines and TFS features such as tasks, jobs, or test reporting.

The instructions in the earlier example demonstrate this approach.

Push an image

After you've built a Docker image, you can push it to a Docker registry or to Azure Container Registry. You can do this by using either the docker push command or by using the Docker task. The Docker task makes the process easier for you because it sets up an authenticated connection to your registry or Azure Container Registry.

If you use Azure Container Registry, make sure that you have pre-created the registry in the Azure portal. You can get the admin user name and password from the Access keys section of the registry in the Azure portal.

Define a variable in your pipeline in the web UI:

  • dockerPassword: Your password for Docker Hub or the admin password for Azure Container Registry.

To push the image to Docker Hub, update your azure-pipelines.yml as follows:

variables:
  dockerId: my-docker-id  # Replace with your Docker ID for Docker Hub or the admin user name for the Azure Container Registry
  imageName: my-image-name  # Replace with the name of the image you want to publish

steps:
- script: |
    docker build -t $(dockerId)/$(imageName) .
    docker login -u $(dockerId) -p $(pswd)
    docker push $(dockerId)/$(imageName)
  env:
    pswd: $(dockerPassword)        # Define dockerPassword in the Variables tab of this pipeline in Pipelines page of web interface

To build and push the image to Azure Container Registry, use the following snippet:

variables:
  dockerId: my-docker-id  # Replace with your Docker ID for Docker Hub or the admin user name for the Azure Container Registry
  imageName: my-image-name  # Replace with the name of the image you want to publish

steps:
- script: |
    docker build -t $(dockerId).azurecr.io/$(imageName) .
    docker login -u $(dockerId) -p $(pswd) $(dockerId).azurecr.io
    docker push $(dockerId).azurecr.io/$(imageName)
  env:
    pswd: $(dockerPassword)        # Define dockerPassword in the Variables tab of this pipeline in Pipelines page of web interface
  1. In your build pipeline, select Tasks, and then add a Docker task to the job that runs your build tasks.

  2. Select the Docker task, and then for Action, select Push an image.

  3. Specify how to connect to your registry in Container registry type and the corresponding service connection properties.

Use Docker Compose

Docker Compose enables you to bring up multiple containers and run tests. For example, you can use a docker-compose.yml file to define two containers that need to work together to test your application: a web service that contains your application and a test driver. You can build new container images every time you push a change to your code. You can wait for the test driver to finish running tests before bringing down the two containers.

If you use Microsoft-hosted agents, you don't have to run any additional steps to install and use docker-compose.

To extend the earlier example to use docker-compose:

  1. Your sample repo already includes a docker-compose.yml file in the docs folder.

  2. Add a Bash step to your build pipeline:

Add the following snippet to your azure-pipelines.yml file.

- script: |
    docker-compose -f docs/docker-compose.yml --project-directory . -p docs up -d
    docker wait docs_sut_1
    docker-compose -f docs/docker-compose.yml --project-directory . down

In the build pipeline, add a Bash task with the following inline script:

docker-compose -f docs/docker-compose.yml --project-directory . -p docs up -d
docker wait docs_sut_1
docker-compose -f docs/docker-compose.yml --project-directory . down

Build ARM containers

When you use Microsoft-hosted Linux agents, you create Linux container images for the x64 architecture. To create images for other architectures (for example, x86, ARM, and so on), you can use a machine emulator such as QEMU. The following steps illustrate how to create an ARM container image:

  1. Author your Dockerfile so that an Intel binary of QEMU exists in the base Docker image. For example, the Raspbian Docker image from Resin already has this.

    FROM resin/rpi-raspbian
    
  2. Run the following script in your build pipeline.

    # register QEMU binary - this can be done by running the following Docker image
    docker run --rm --privileged multiarch/qemu-user-static:register --reset
    # build your image
    docker build -t $(dockerId)/$(imageName) .
    

Troubleshooting

If you can build your image on your development machine, but you're having trouble building it on Azure Pipelines or TFS, the following solutions might help:

  • Check that you are using the correct type of agents - Microsoft-hosted Linux or Microsoft-hosted Windows - to mimic the type of container images you build on your development machine.

  • If you use Microsoft-hosted agents to run your builds, the Docker images are not cached from build to build because you get a new machine for every build. This will make your builds on Microsoft-hosted agents run longer than those on your development machine.

If you can build your image on your development machine, but you're having trouble building it on Azure Pipelines or TFS, check the version of Docker on the agent. Ensure that it matches what you have on your development machine. You can include a command-line script docker --version in your build pipeline to print the version of Docker.