Progressive experimentation with feature flags

As DevOps teams shift to an agile methodology that focused on the continuous delivery of features, the need to control disclosure of new features becomes increasingly important. Whether the goal is to restrict access to those new features for marketing purposes or in order to limit the audience for testing in production, feature flags are a great solution.

Decoupling deployment and exposure

Feature flags are settings a team can define that indicate whether a given set of features is visible in the user experience and/or invoked within the functionality. This allows the team to build and deploy those features as part of the ordinary development process, but to avoid having those features available for broad access. They offer a convenient way to decouple the deployment of features from their exposure.

Flags provide runtime control down to individual user

Flags also provide granular control all the way down to the individual user. When it's time to enable a feature, whether for one user, a small group, or everyone, the team can simply change the feature flag in order to light it up without having to redeploy.

The scope of a feature flag will vary based on the nature of the feature and the audience. In some cases, a feature flag will automatically flip the functionality on for everyone. In other cases, a feature will be enabled on a user-by-user basis. Teams may also opt to use feature flags to turn on an option for users to enable a feature, if they so desire. There's really no limit to the way the feature flags are implemented.

Support early feedback, experimentation

Feature flags are a great way to support early experimentation. Some features can have rough edges early on, which may only be interesting to the earliest of adopters. Trying to push those not-quite-ready features on a broader audience could product dissatisfaction. But the benefit of gathering feedback from those willing to deal with an in-progress feature is invaluable.

Quick off switch

Sometimes it's very helpful to be able to turn something off. For example, suppose a new feature isn't working the way it was intended, and there are side effects that are causing problems elsewhere. Feature flags are a great way to quickly turn off the new functionality in order to roll back to trusted behavior without having to redeploy. While feature flags are often thought of in terms of user interface features, they can just as easily be used for changes in architecture or infrastructure.

Standard stages

There is a standard rollout process Microsoft uses to turn on feature flags. There are two separate concepts: rings are for deployments, and stages are for feature flags. Learn more about rings and stages.

Stages are all about disclosure or exposure. For example, the first stage could be for a team's account and the personal accounts of members. There's likely a lot of stuff running on a service right now that users wouldn't see because the only place flags are turned on is for this first stage. This allows a team to fully use and experiment with it. Once the team signs off, select customers would be able to opt into it via the second stage of feature flags.

Opt in

It's a good practice to allow users to opt in to feature flags when feasible. For example, the team may expose a preview panel associated with the user's preferences or settings.

Opt in preview pane

Use flags with telemetry

With feature flags, teams have a way to incrementally expose updates. But to fully gauge the readiness for broader exposure, teams must continuously monitor the right metrics. These metrics should include usage behavior, as well as the impact of the updates on the health of the system. It's important to avoid the trap of assuming everything is okay just because nothing bad seems to be happening.

A feature flag example

Consider the example below. The team added a couple buttons here for Cherry-pick and Revert in the pull request UI. These were deployed using feature flags.

Pull request UI example

Defining feature flags

The first feature exposed was the Revert button. In the case of this product, the solution uses an XML file to define all of the feature flags. In this case, there's one file per service. That ensures that if a team ends up with their section getting really long, it's clear it's time to clean them up. They'll delete old flags because there's a natural motivation to control the size of that file.

<?xml version="1.0" encoding="utf-8"?>
<!--
  In this group we should register Azure DevOps specific features and sets their states.
-->
<ServicingStepGroup name="AzureDevOpsFeatureAvailability" … >
  <Steps>
    <!-- Feature Availability -->
    <ServicingStep name="Register features" stepPerformer="FeatureAvailability" … >
      <StepData>
        <!--specifying owner to allow implicit removal of features -->
        <Features owner="AzureDevOps">
           <!-- Begin TFVC/Git -->
           <Feature name="SourceControl.Revert" description="Source control revert features" />

Having a common server framework makes for a lot of reuse and economies of scale across the whole team. Ideally, the project will have infrastructure in place so that a developer can simply define a flag in a central store and have the rest of the infrastructure handled for them.

Checking feature flags at runtime

The feature flag used here is named SourceControl.Revert. Here's the actual Typescript from that page that illustrates the call for a feature availability check.

private addRevertButton(): void {
 if (FeatureAvailability.isFeatureEnabled(Flags.SourceControlRevert)) {
     this._calloutButtons.unshift(
         <button onClick={ () => Dialogs.revertPullRequest(
             this.props.repositoryContext,
             this.props.pullRequest.pullRequestContract(),
             this.props.pullRequest.branchStatusContract().sourceBranchStatus,
             this.props.pullRequest.branchStatusContract().targetBranchStatus)
         }
         >
             {VCResources.PullRequest_Revert_Button}
         </button>
        );
     }
}

The example above illustrates usage in TypeScript, but it could just as easily be accessed from C#. The code checks to see if the feature is enabled and, if so, renders a button to provide the functionality. If the flag isn't enabled, then the button is skipped.

Controlling a feature flag

A good feature flag platform will provide multiple ways to manage whether a given flag is set. Typically, there are usage scenarios for the flag to be controlled via PowerShell and web interface. For PowerShell, all that really needs to be exposed are ways to get and set a feature flag's status, along with optional parameters for things like specific user account identifiers (if applicable).

Controlling feature flags through web UI

Below is an example of using the web UI exposed for this product by the team. Note the feature flag for SourceControl.Revert. There are two personal accounts listed here: hallux and buckh-westeur. The state is set for hallux, which happens to be in North Central, and cleared for the other account in West Europe.

Controlling feature flags through web UI

The nature of the feature flag will drive the way in which the features are exposed. In some cases, the exposure will follow a ring and stage model. In others, users may opt in through configuration UI, or even by emailing the team for access.

Considerations for feature flags

Most feature flags can be retired once a feature has been rolled out to everyone. At that point, the team can delete all references to the flag in code and configuration. It's a good practice to include a feature flag review, such as at the beginning of each sprint.

At the same time, there may be a set of feature flags that are persistent for a variety of reasons. For example, the team may want to keep a feature flag that branches something infrastructural for a period of time after the production service has fully switched over. However, keep in mind that this potential codepath could be reactivated in the future during an explicit clearing of the feature flag, so it needs to be tested and maintained until the option is removed.

Feature flags and branching strategy

Feature flags enable development teams to include incomplete features in main without affecting anybody else. As long as the codepath is isolated behind a feature flag, it's generally safe to build and publish that code without side effects impacting normal usage. But if there are cases where a feature requires dependencies, such as when exposing a REST endpoint, teams must be mindful of how those dependencies can create security or maintenance work even without the feature being exposed.

Feature flags to mitigate risk

Sometimes new features have the potential to introduce destructive or disruptive changes. For example, the product may be undergoing a transformation from a wide database schema to a long one. In that scenario, the developer should create a feature branch for a small amount of time. They then make the destabilizing changes on the branch and keep the feature behind a flag. A popular practice is for teams to then merge changes up to main as soon as they're not causing any harm. This wouldn't be feasible without the ability to keep the unfinished feature hidden behind a feature flag.

Feature flags help work in main

Working in main is a good way to tighten a DevOps cycle, provided it follows the common sense practices discussed in the Develop phase. When combined with feature flags, developers can quickly merge their features upstream and push them through the test gauntlet. This ensures that quality code quickly gets published for testing in production. After a few sprints, developers will quickly recognize the benefits of using feature flags and being using them proactively.

How to decide whether to use a feature flag

The feature teams own the decision as to whether they need a feature flag or not for a given change. Not every change requires one, so it's a judgment call for a developer when they choose make a given change. In the case of the Revert feature discussed earlier, it was important to use a feature flag because it meant that the team could put the feature out in production with controlled exposure. Allowing teams to own key decisions about their feature area is part of enabling autonomy in an effective DevOps organization.

Build vs. buy

While every team has the option of building their own feature flag infrastructure, it's generally recommended that they adopt a platform like LaunchDarkly, if possible. It's preferable to invest in building features instead of rebuilding feature flag functionality.

Next steps

Learn more about using feature flags in an ASP.NET Core app.