Volume 32 Number 1
Automate Complex Deployments with Release Management
By Kraig Brockschmidt | January 2017
The happy phrase, “Ship it,” declares that whatever software you’re producing, such as mobile apps and their associated back-end services, is now ready for deployment to customers or, as it’s said, “into production.” But how, exactly, does software get to that point? In my last article in this series (msdn.com/magazine/mt742871), I explored how Build produces the artifacts that feed into the release pipeline. That bundle of artifacts, called the “release,” is what then undergoes any number of tests and other processes, as shown in Figure 1, on its journey to production. The bulk of DevOps activities, in fact, involve the deployment of a release into any number of environments where certain tests can be run, and shepherding that release along to the next stage of the pipeline.
Figure 1 Managing a Release Happens Between Build and Public Deployment
A release process potentially involves a large number of different tests that don’t necessarily run simultaneously, and which might also require different machines and devices. It might also involve direct approvals by human beings who eat lunch and leave their desks for evenings and weekends. As a result, the time it takes for a release to get through the entire pipeline could easily be a matter of days. Meanwhile, your dev team continues to work their backlog for subsequent releases, committing code to the source repository, thus triggering more builds that produce artifacts that feed into the appropriate pipeline.
Managing the flow of all these artifacts through multiple pipelines can easily become a complex and demanding task, thus the Release Management tools in Visual Studio Team Services (VSTS) and Team Foundation Server (TFS) are an essential part of the Microsoft DevOps stack.
On the surface, release management looks mostly like administrative support, and thus might not be as technically interesting as other parts of the Microsoft DevOps stack. But like everything else in DevOps, release management is fundamentally a practice that begins as a list of steps you can perform manually, in this case to validate that the artifacts from the Build/CI stage are ready to deploy to subsequent environments. Once you have those steps clearly defined—when to deploy artifacts to a particular environment, which tests to run and the criteria for moving the artifacts to the next stage—you can then use tools to incrementally automate those steps.
In many ways, managing a release is a lot like setting up an automated build, except that the output of release management is the deployment of build artifacts to places where the right people can get at them. Ultimately, deployment is how the value contained in your source code is actually delivered to your customers, which is, of course, the whole point of the software development process!
Environments, Pipelines and Managing Complexity
Although the diagram in Figure 1 shows only two QA stages—one internal and one external—there can really be any number of stages. This is because different forms of testing require deployment into specific environments where those tests can be carried out. An environment is simply a particular configuration of hardware, software and data that’s suitable for the desired testing or usage scenarios. Here are a few that are commonly discussed in the context of DevOps:
- Machines that are capable of doing unit tests, integration tests, and UI tests are typically part of an automated test environment, usually using mock data, test services, and servers configured for different load and stress tests.
- A similarly configured manual test environment is generally where a dedicated test team does its work.
- A staging environment is used for deploying apps and services for full-scale pre-production tests (such as upgrade tests), and includes deployment to alpha- and beta-test customers. This environment might draw upon live data and services in a read-only manner, or use staging versions to fully simulate live activity. This is also where you can test your crash and telemetry reporting systems and make sure they’re generating the data you want.
- Your public production environment, finally, uses live public data and services, of course, and is where you collect the live telemetry data that will ultimately feed into subsequent releases.
You can, of course, define whatever environments you want according to your validation needs. Whatever the case, how build artifacts travel through these stages and environments—and what tests are applied where—is again what defines a release pipeline, and you can have any number of different routes in operation for different purposes. With mobile apps and back-end services, you’ll have multiple deployment targets all along the way—different emulators or device farms, staging servers and so on. You might also have releases go through only part of the pipeline just for testing purposes. With a public release, the “rollout track” feature of Google Play (bit.ly/2b7lh8j) also allows you to release a new version of an app to a limited percentage of customers—which is essentially another production environment—thus, you might have a separate release pipeline for that track.
What Is a Release?
The term “release management” clearly implies that a “release” is being managed in some way. A release is a specific bundle of build artifacts you intend to move through a series of validations and deploy to one or more environments. Those artifacts (along with other information and metadata) always travel through the pipeline as a unit, regardless of any subsequent changes to the source repository and the builds they might trigger, because every set of build artifacts is unique and stamped with a full version number like 220.127.116.11, where the last number is the build. Such versioning buys you complete end-to-end auditing, which is the ability to trace everything that happens within the release pipeline back to the exact build, and thus to an exact set of changes in the source repository.
In DevOps parlance, then, “starting a release” means feeding a worthy unit of build artifacts into the release pipeline. Figure 2 illustrates what the process might look like: Using the project backlog for communication, the team applies a series of validation checks, irrespective of automation, that either reject the release or allow it to proceed to the next stage, eventually to reach production.
Figure 2 A Unit of Build Artifacts Flows Through Various Stakeholders on Its Way to Production
As noted before, you can certainly have any number of testing and staging environments, and even multiple production environments (as when using the Google rollout feature). Because a release process can take considerable time, you might choose to deploy to production only once a week or once a month. In the meantime, you’ll still want to run other releases through some part of the pipeline, perhaps deploying to beta testers each week to get feedback for each monthly production release. You’ll probably also want daily builds to go to the test manager, and likely want every build to go through a series of quick, automated tests to provide ongoing feedback to the dev team. In short, the frequency of releases can change as you go up and down the diagram in Figure 2.
Thus, release management is something that can start simple and grow from there. The simplest form of this practice, in fact, is something you’ve likely done already: You build an app package, deploy it to a device (staging), and play with the app to make sure it works as expected. Then you upload the package to a store, hit “Publish”—and presto! You’ve just done a manual release process to put your app into production.
Continuous Deployment as a Culture
As I explained in the first article in this series, (msdn.com/magazine/mt767694), all DevOps processes begin with being completely clear about what needs to happen at every stage along the release pipeline, automated or not. You should be able to describe all your processes in a simple document, such that every step could be done manually. Then you’re ready to apply automation to reduce costs, improve reliability and consistency, and increase frequency of testing and deployment.
A primary goal within DevOps is to have every new release of an app or service—including releases with only minor changes—flow as quickly as possible from the source repository to customers. A completely automated flow is called continuous deployment (CD), which goes hand-in-hand with continuous integration (CI): Every commit to your repository triggers a new CI build, and every successful build—which produces a new bundle of artifacts with a specific version number—triggers a new automated release process. That release process then carries out all the necessary validations before deploying that bundle to production. CD, in short, means continually delivering value to your customers at the lowest cost, with minimal (if any) human intervention along the release pipeline.
Realize, however, that although CD optimizes the release pipeline between the Build/CI stage and deployment to production, it still requires vigilant effort by people to make it work:
- Your team needs strong code-review processes to prevent poor code from being committed to the repository to begin with.
- Your team must have high confidence that automated tests—which people create—are catching most defects and preventing them from reaching customers.
- Because no suite of tests is perfect, some defects will get through to production, so your team must actively monitor crash reports, telemetry and direct customer feedback in your production environment.
- Your team must be committed to quickly triaging and prioritizing issues and feeding them into the dev backlog so that corrections quickly get into subsequent releases.
- Issues also identify gaps in your code-review process and test coverage, thus driving improvements in both.
In short, CD isn’t just a matter of automating your release pipeline: CD is a culture of using feedback to constantly improve how you’re delivering value to customers.
As an example, the documentation for Microsoft Azure, found on azure.microsoft.com/documentation, is managed within an open source repository on GitHub, github.com/Azure/azure-content. There’s a full CI/CD system in place such that any changes accepted into the repository through pull requests quickly get out to production. Pull requests, however, are carefully scrutinized by the Azure content team at Microsoft, which prevents incorrect or inappropriate edits from getting into the repository at all. Accepted changes then pass through the automated CI/CD pipeline that applies a variety of validation tests (such as catching incorrect formatting and broken links), and then publishes the content to the live site. The team then monitors telemetry reports from Application Insights along with customer comments, using that information to improve the content, improve the validations tests, and improve the review process itself.
Release Management in VSTS
In my last article, I looked at setting up automated builds with VSTS by taking a known build process, creating a build definition from it and feeding that definition to a build agent that’s capable of performing the necessary tasks to produce a bundle of artifacts with a specific version number.
Release Management in VSTS is a similar process: You create a release definition that specifies how that bundle is to be deployed to different environments, and the tests and validations that are to be applied to it. That release definition is then fed to a release agent—a suitably configured machine—for processing (release agents are managed in the same way as build agents, see bit.ly/2coBQxx). That said, there are a number of significant differences between build and release definitions:
- A build definition produces testable and deployable artifacts; a release definition guides the actual deployment to an environment and the running of tests.
- A build definition always works from a single-source repository; a release definition can draw from any number of build definitions to collect the artifacts for a release.
- You can often set up a fully automated build in a matter of hours; a fully automated release pipeline happens over time because it takes much more effort to create the underlying automated tests and work out the details for approvals and sign-offs.
- A typical build completes in a matter of minutes; a full release process, with multiple environments and manual approval steps, will take much longer. As a result, you’ll be monitoring and auditing many releases in your pipeline at different stages.
- A release definition supports both pre- and post-deployment approvers, with which you inject manual control at either end of a release step, as well as explicit manual intervention tasks. A simple example is using one or more pre-deployment approvers before going to production. VSTS also lets you use a group as an approver, such that only one person in that group needs to sign off. In general, you assign an approver any time you want a human being to be involved in the release process, which also results in leaving an audit trail.
As an example, let’s say I have a Xamarin app and back-end code in a Git repository on VSTS, with four build definitions that produce the back-end artifacts and app packages for iOS, Android and Windows. A simple release process—echoing the stages shown in Figure 2—might be as follows, where a failure at any point in the pipeline will cancel the release:
- Run unit and integration tests on a successful build.
- Test environment
- Deploy the app to Xamarin Test Cloud to run tests on physical devices.
- Deploy the back end to a local load-test server.
- Staging environment
- Deploy the app to beta testers using HockeyApp.
- Deploy the back end to a staging server (used by beta testers) running on Azure App Service.
- Require sign-off by an approver who reviews beta-tester feedback from and evaluates the readiness of the release.
- Production environment
- Deploy the app to Google Play, Apple App Store and Windows Store.
- Deploy the back end to the production Azure App Service.
Note, once again, that this process says nothing about automation—it simply describes the steps involved in making a release. Because of this, I can start building release definitions with simple steps like deployment and approver sign-offs. Then, as I get various tests put together, I can incrementally add steps to increase automation. (By the way, this process is essentially what was set up for the MyDriving project [aka.ms/iotsampleapp], although it ends at step 3 because the app is distributed only through HockeyApp.)
Technically speaking, you can include testing and deployment steps directly in a VSTS build definition. This is sufficient when deploying to and testing in only a single environment. When multiple environments, approvers, and other release-specific steps are involved, however, you’ll need the fine-grained control of Release Management.
Walk-Through: Deploying to Successive Azure Environments
Now I’ll do a walk-through of the main process of using Release Management, for which I’ve created a Xamarin app and a Node.js service from templates in Visual Studio. In VSTS, I created a Team Project called MSDN Magazine Dec 2016, and added the projects to its source repository. I set up four build definitions to generate suitable artifacts that I can follow through a release pipeline, even if those artifacts don’t do anything interesting. (To create build definitions for different back-end project types, see “Build Your App” [bit.ly/2cGbq7W] in the VSTS documentation, which ensures that you create deployable artifacts for Azure and demonstrates deployment tasks in build definitions.)
Because I have four distinct bundles of artifacts going to different targets, I’ll eventually need four release definitions, but here I’ll start with just the back end. On the Team Services portal, I navigate to the Team Project, select the Release tab, and click + New definition. As with Team Foundation Build, this brings up a dialog with (as of this writing) just a few Azure-related release templates—for any destination other than Azure, including app stores, just start with an empty definition. For my back end, I use the Azure Website Deployment template and click Next. This brings up a configuration dialog in which I select my back end’s build definition as the source of the artifacts (you can also use Jenkins as a source). I also select an agent queue and set an option for continuous deployment, which I’ll return to later.
VSTS then opens the release definition in the editor shown in Figure 3 (by default there will be just a single environment). The editor is organized, left to right, into environments, tasks and details for the task. The general workflow is to create an environment first, and then populate it with appropriate tasks for that environment. Because environments often have similar steps, you can create the tasks for one environment and then clone that environment to save time. The + Add Environment button gives you this option. (You can also create meta-tasks, in which you group a sequence of tasks together to create a new single task, parameterized with variables, that can be used in build and release definitions. For a full walk-through, refer to the VSTS documentation at bit.ly/2c1X3fP.)
Figure 3 Editing a Release Definition on Visual Studio Team Services
As with build definitions, any tab that needs attention is highlighted in red—in Figure 3, I need to identify the target Azure App Service for the deployment. In my Azure account, then, I create App Service instances called kraigb-MSDN1216-test, kraigb-MSDN1216-staging and kraigb-MSDN1216-prod. I then return to VSTS and click the Manage link on the right-hand side next to Azure Subscription (Classic), which takes me to the Services tab on the Team Project control panel. I select + New Service Endpoint, select Azure Classic, and get a dialog in which I enter connection information. The easiest way to work with this is to select the publish settings file link in the dialog that goes to the Azure portal and generates a text file for download, from which you can copy-paste values into VSTS.
Having established this connection, I return to the release definition, refresh the subscription list and select my new connection. From there I can refresh the Web app location and Web app name controls to select the App Service instance I want.
Because most of the connection information for my Test environment is the same for my Staging and Production environments, I can clone Test twice and rename the copies. When I make the clone for Production, though, I also set myself as a pre-deployment approver and check a box so I get an e-mail when a release is waiting. By doing this, I’ve injected a pause between the Staging and Production environments.
Starting a Release
I now have a basic deployment pipeline between three environments: test, staging, and production, where the first two deployments happen automatically and the third is subject to manual approval. To start a release manually, I click + Release in the release definition and select Create Release. This prompts me with the dialog in Figure 4 in which I select the build to use (from any successful build that’s still in VSTS), and where I can also control the deployment chain between the environments. Notice how, for the Test environment, deployment happens automatically upon creation of the release, which is when I click the Create button. Staging and Production, similarly, have automatic deployment depending on the previous environment, subject, of course, to the success of any other release steps in those environments (such as tests), and any necessary approvals.
Figure 4 Starting a New Release Manually by Selecting a Build and Setting Deployments
Once I’ve clicked Create and the release begins, I can navigate into that release to check on its progress, as shown in Figure 5.
Figure 5 Examining the Status of a Release in Progress
I’ll share that when I first created this release pipeline, using a Node.js back end, my build definition didn’t create any actual artifacts, and so the release failed. I checked this by navigating to my most recent build and clicking the Artifacts tab on its summary page. Once I added the necessary Gulp task to produce some real output, as instructed by the guide on bit.ly/2c1XhTY, those artifacts were in place.
In my simple release pipeline, I haven’t set up any automated tests so deployments to Test and Staging happen quickly in succession, and I can go to those Web sites and see the results. Before anything is deployed to Production, however, the release tells me that “A pre-deployment approval is pending for ‘Production’ environment. Approve or Reject,” as you can see in Figure 6. I also get an e-mail alerting me to the approval, with a link to that same release page in VSTS. There I click the Approve or Reject link to indicate my decision, add comments, defer the deployment to a later time or reassign the approval to someone else. In this case I click Approve, the release completes, and I’m able to go see the deployment on the kraigb-1216-prod App Service instance.
Figure 6 Approving or Rejecting a Manual Approval Step
Figure 6, as you can see, shows the status for a single release. By clicking on the name of the release definition in the upper right, I can see a status summary page of all releases that are still being retained within VSTS. This is where you can visually monitor the progress of every release in your pipeline for this definition, and a similar view is available by clicking All Release Definitions in the navigation tree on the left side of the portal (not shown).
Setting up continuous deployment means automatically starting a release after a successful build using the artifacts produced by that build. To do this, I edit the release definition and click on the Triggers tab, where there are options for Manual, Continuous Deployment and Scheduled releases. When I click on Continuous Deployment, I then select the source of the artifacts, which is my build definition for the back end, and save the release definition.
I can now go all the way out to Visual Studio, make a change in the back-end code, and commit it to the repository. This triggers a new build on VSTS because I checked the Continuous Integration box in the back end’s build definition. Once that build succeeds, it automatically triggers a new release because I have Continuous Deployment checked in the release definition. Thus, with both CI and CD options, I’ve set up the full release pipeline between changes in the code and deployment to production, subject only to the manual pre-deployment approval that I specified for the Production environment in the release definition.
Although this pipeline is simple, I now have a solid foundation on which I can build out additional steps, such as running any number of tests, simply by adding more tasks to the appropriate environment in the release definition, and setting up any necessary pre- and post-deployment approvals. I can also add new environments to divide the pipeline into even more distinct stages. But no matter how many tasks or environments you add, the basic process will be the same as you’ve seen here.
Walk-Through: Releasing the App
With the back-end release definitions in place, I can now create the three release definitions for the iOS, Android and Windows builds of the Xamarin app. The process is very much the same as with the back end, except that I start with a blank release template and select the platform-specific build definition. In the release definition editor, I won’t have any tasks by default, so once I set up the environments I want (such as Test, Pre-Launch and Launch, to show you can use any names you want), I click on + Add task and select one from the dialog that appears.
In the case of the Android app, my deployments are as follows:
- Test: Use a build task to compile any test code, then use a Xamarin Test Cloud task to deploy the .apk file and test code to run those tests automatically on devices I’ve selected. (Of course, a Test Cloud account is necessary to use the service.)
- Pre-Launch: Use the HockeyApp task from the VSTS Marketplace, which I first install so it appears in the Add task dialog. I will have also created an account with HockeyApp and used its services to set up my list of pre-launch customers.
- Launch: For Android, I install the Google Play task from the Marketplace and select it from the Add task dialog (where you’ll see Release, Promote and Increase Rollout tasks separately). This means, of course, that I’ve set myself up as a developer with Google.
For my iOS and Windows release definitions, I can still use Xamarin Test Cloud and HockeyApp, but those platforms don’t provide for automated deployment to their respective stores. In these cases, my Launch environment doesn’t have any tasks. Instead, I’ve simply assigned a pre-deployment approver to that environment. That person is responsible for evaluating feedback from the pre-launch testers and can then go out to the appropriate portals to upload the production app.
Pre-Launch Distribution with HockeyApp
HockeyApp (hockeyapp.net) is a powerful DevOps service for iOS, Android, and Windows apps both before and after a launch. Post-launch, HockeyApp provides crash reporting, usage data and user feedback, as will be covered in a later article. Pre-launch, HockeyApp provides these same services along with the ability to quickly and easily deploy test apps to any number of testers, no matter what devices they’re using and no matter where they are in the world. It also lets you organize and manage your testers in a variety of ways so you can easily control which groups get which release, and when.
Pre-launch distribution (bit.ly/2cxruZ8) works through the HockeyApp client that testers install on their devices. When you have a new test release ready, you upload it to the HockeyApp portal either manually or through a deployment step in VSTS. Your testers then receive an e-mail saying that the new release is available, and the HockeyApp client shows available releases that can be installed using the side-loading capabilities of the various mobile platforms.
In this series of articles, I’ve now looked at source control, build, and release management, through which you can create a full release pipeline across any number of environments that will accommodate whatever additional testing and approval steps you might require. This completes all the core connections between your source code and your customers. What remains, now, is to understand how to listen to those customers through monitoring of both apps and back-end services, and then to learn more about the variety of testing options you can employ, including Xamarin Test Cloud.
Thanks to the following technical expert for reviewing this article: Alex Homer
Alex Homer is a Microsoft technical author who has eschewed the delights of Redmond to work from home in the glorious Derbyshire Dales in England. Miscellaneous characters in his articles are contributions from his two cats.