How Microsoft plans efficient workloads with DevOps
Microsoft is one of the largest Agile companies in the world. In our years of experience we have developed a DevOps planning process that can be scaled from the smallest projects up through massive efforts like Windows. This article details many of the practices implemented and lessons we learned when planning major products across the company.
Before vs After
Below you can find a summary of the changes our development teams enacted as we made the leap to apply Agile practices to our workflow.
|4-6 month milestones||3-week sprints|
|Horizontal teams||Vertical teams|
|Personal offices||Team rooms and remote work|
|Long planning cycles||Continual Planning and learning|
|PM, Dev, Test||PM, Design, and Engineering|
|Yearly customer engagement||Continual customer engagement|
|Feature branches||Everyone in
|20+ person teams||8-12 person teams|
|Secret roadmap||Publicly shared roadmap|
|Bug debt||Zero debt|
|100 page spec documents||Specs in PPT|
|Private repositories||Open source/Innersource|
|Deep organization hierarchy||Flattened organization hierarchy|
|Success is a measure of install numbers||User satisfaction determines success|
|Features shipped once a year||Features shipped every sprint|
There are four key changes that we decided to implement to really make our development and shipping cycles efficient and healthy.
We changed our culture
"Culture eats strategy for breakfast." -- Peter Drucker
We've found that the key motivating factors for people are autonomy, mastery, and purpose. At Microsoft, we are looking for ways to bring those factors to our PMs, developers, and designers so that they feel empowered to build the right things.
Two items we heavily consider for our approach are alignment and autonomy:
- Alignment happens from the top down. It ensures that individuals and teams understand how what they are responsible for lines up with broader business goals.
- Autonomy happens from the bottom up. It ensures that individuals and teams come into work every day feeling like they have a big impact on day-to-day activities and decisions.
There is a delicate balance between the two. Too much alignment drives a negative culture where people punch in and punch out because that's what the "manual" says to do. Too much autonomy and there's no structure or direction, leading to inefficient decisions and plans.
We changed our focus from individuals to teams
We started thinking about teams more than individuals. We organized people and teams into three groups: PM, design, and engineering.
- PM is responsible for defining what we're building and why we're building it.
- Design is responsible for designing what we're building.
- Engineering is responsible for how we're building it and ensuring we build it with quality.
Key characteristics of our teams
- Cross discipline
- 10-12 people
- Clear charter and goals for 12-18 months
- Physical team rooms
- Own features in production
- Own deployment of features
Makeup of teams
Before, teams would be horizontal. They would either cover all UI, all data, all API, etc. Now, we define teams that own their area end-to-end in the product. We implement strict guidelines in certain tiers to ensure uniformity across the product among teams.
We strive for vertical teams:
Every ~18 months we practice what we call the "yellow sticky exercise". This is an activity in which developers are able to choose which areas of the product they're interested in and would like to work on for the next couple of planning periods. This exercise provides the teams with autonomy. They are able to choose what they work on, which also helps the organization with alignment as it makes sure we have balance among the teams.
About 80% of the people in this exercise end up going back to the team they currently work for, but they go back feeling empowered that they had a choice.
Transparency and accountability
At the end of every sprint, every team sends a mail saying what they've accomplished in the previous sprint, and what they plan on doing in the next sprint.
We changed how we plan and learn
"Plans are worthless, but planning is everything." -- Dwight Eisenhower
Our planning breaks down as follows:
- Sprints (3 weeks)
- Plans (3 sprints)
- Seasons (6 months)
- Strategies (12 months)
Engineers and teams are responsible mostly for sprints and plans. Leadership is mostly responsible for seasons and strategies.
We've found that this planning structure helps us maximize our learning while we do our planning. We're able to get feedback, deploy, find out what customers want, and actually implement those things quickly and efficiently.
We created new ways to stay healthy
Before, we would let code bugs build up until the end of a code phase (the "code complete"). We would then discover them, work on fixing them, rinse, and repeat. This created a "roller coaster" of bugs, and as the number of bugs dropped, so did team morale as they did nothing but work on bug fixes instead of implementing new and fun features.
Now, we implement what we call a bug cap. A bug cap is calculated by the following formula:
# of engineers x 5 = bug cap
If a team's bug count exceeds the bug cap at the end of a sprint, they stop working on new features until they are back down under their cap. This is a version of paying down debt as we go.
We shielded teams from distractions
Teams have come up with their own way to provide focus and assist with an interrupt culture in the form of bugs and live-site incidents.
Teams will self-organize for each sprint into two distinct crews: Features (F crew) and Customer (C crew). The F-crew works on committed features and the C-crew deals with all live site issues and interruptions. The rotating cadence is established by the team and it allows for members to plan activities outside of work much easier. Learn more about using the multi-crew model to build productive teams.
Creating clarity through objectives and key results (OKRs)
We have found that teams are most effective when they have clarity on the goals the organization is trying to achieve. This can most effectively be achieved through use of objectives and key results.
- Objectives define what is to be achieved. They are significant, concrete, action oriented, and (ideally) inspirational statements of intent for a team. They are the big ideas and not the actual numbers.
- Key Results measure progress to achieve objectives. They are the quantifiable outcomes that that indicate success against the objective in a specific time period so progress can be evaluated.
OKRs should reflect the best possible results and not just the most probable results. Leaders need to be ambitious and not cautious. Pushing teams to pursue challenging key results drives acceleration against objectives and prioritizes work that moves towards larger goals.
Key results are only as useful as the metrics on which they are based. Prefer leading indicators that focus on change. Over time these metrics will build a working picture of product acceleration or deceleration. Consider using some of the following:
- Change in monthly growth rate of adoption
- Change in performance
- Change in time to learn
- Change in frequency of incidents
As a matter of practice, teams should avoid metrics that don't accrue value toward the objective. While they may have their place elsewhere, here are some examples of deceptively unhelpful metrics:
- Accuracy of original estimates
- Completed hours
- Lines of code
- Team capacity
- Team burndown
- Team velocity
- Number of bugs found
- Code coverage
Adopting an OKR framework can help teams perform better for a variety of reasons.
- Every team is aligned on the plan.
- Teams focused on achieving outcomes rather than completing activities.
- Every team is accountable for efforts on a regular basis.
OKRs may also exist at different levels of a product. For example, there can be top-level product OKRs, component-level service OKRs, and team-level OKRs. Keeping them aligned should be relatively easy, especially if set top-down. Any conflicts that arise are valuable early indicators of misalignment within the organization.
An OKR example
Objective: Grow a strong and happy customer base.
- Increase external NPS from 21 to 35.
- Increase docs SAT from 55 to 65.
- New pipeline flow has an Apdex score of 0.9.
- Queue time for jobs is 5 seconds or less.
Learn more about measuring business outcomes using OKRs.
Four key takeaways
- Take agile science seriously, but don't be overly prescriptive. Agile can become too strict, let it grow like a mindset or culture.
- Stop celebrating activity and start celebrating results. Lines of code shouldn't outweigh functionality deployments.
- You can't cheat shipping. Until you get into the mindset of updating services/products at the end of sprints, you won't find all of the work that needs to be done. Shipping every Sprint helps establish a rhythm and cadence.
- Build the culture you want and you'll get the behavior you're looking for.