Release: Communication, Training, and Deployment

Applies to: Agile

Authors: Greg Smith and Ahmed Sidky

Get this book in Print, PDF, ePub and Kindle at manning.com. Use code “MSDN37a” to save 37%.

This topic contains the following sections.

• When to Release
• Final Testing
• Preparing Support Groups and Processes
• Copyright

At this point you may be thinking, “Why are you talking about delivery?” Isn’t delivery easy in an Agile environment? After all, you make sure each iteration is production ready. If each iteration is deployable, deploying should be a piece of cake.

The fact that your code is in a deployable state is a plus; but in reality, releasing code can be a major effort. In some environments, releasing code can be more complicated than building code.

For the purposes of this chapter, we’ll discuss delivery from the perspective of a complete project or release. We’ll look at deploying a set of features with a synchronized delivery to the production environment. This type of deployment is common for teams that release on a synchronized schedule, or when you deliver a project with several interdependent features. This is the case with the Auctionator for Acme Media: several auction features work together to bring value to the end user, so they must be deployed at the same time.

When you build code, you’re working with a small group, and it’s easier to manage the project. When you release the code, you’ll affect end users, customers, marketing, support, stockholders, stakeholders, training teams, and help desks. In some companies, a compliance group must be engaged to release the code.

In this chapter, we’ll go through all the steps that Acme Media (and potentially you) may go through when deploying a project. You’ll complete your testing, prepare support groups, train end users, turn on your marketing plan, and deploy the code into the production environment.

When to Release

A production release can be triggered by a deadline, by a predetermined release schedule, or when there is enough value. You may also release to production to perform a beta test of a product or application.

Let’s take a moment to look at the various constraints that may drive the release of your product.

To Support a Constraint

Four common project constraints are customer deadlines, regulatory requirements, resource limitations, and competitor-driven deadlines. Let’s look at each.

Supporting a Customer Contract

Although contracts and Agile development aren’t the best of partners, you may still find yourself working with a customer who requires a contract. In these instances, you’ll still use your tradeoff matrix to help the customer understand the compromises required to support the fixed date. You and your team will focus on delivering to the date.

One of the Agile principles is Customer collaboration over contract negotiations. If you follow this principle, your customer will pay you iteratively as you deliver value and as they develop a better understanding of their needs.

When we coach Agile teams, we emphasize this point. Our clients agree with the principle but often tell us that they work in a contractual environment and that there is no easy way to leave that environment. They can collaborate with the customer during the project, but there is still a looming contract deadline.

We’ve seen this in our own work experience. In our early years of work, a sales or marketing group would make an outlandish commitment to a customer to secure a sale; then they would come visit our development team and tell us about the miracle we had to pull off.

In recent years, we’ve worked on contractual projects, but the sales teams have become more collaborative and involve the development team in the estimation process.

This helps the team buy into the project and also minimizes over-commitment and the potential for creating a death-march project (a project destined to fail).

Supporting a Regulatory Deadline

Projects are often driven by the need to meet a compliance deadline. In recent years, many companies have scrambled to deliver projects that support the Sarbanes-Oxley Act. In 1999, we all scrambled to deliver systems that supported Y2K.

In the past five years, we’ve worked on compliance projects for Basel II (control over financial reporting and supporting processes), the Health Insurance Portability and Accountability Act (HIPAA; integrity and confidentiality of health records), and Hazard Analysis and Critical Control Point (HACCP). Regulatory compliance is a project reality for many teams.

Supporting a Financial or Resource Deadline

Larger projects require a commitment of capital funds, and the fund commitment is usually tied to a budget. In these instances, a project must be completed before the funding expires.

You may also find that you have a budgeted amount of time for the project team. This is usually true of consulting resources or shared resources within your company.

Meeting a Deadline from your Competition

One of the main strengths of Agile is helping you quickly deploy functionality to catch and overcome a competitor. You’ve seen this in action with the Acme Media case study. Acme Media needed to catch its competitors before the company’s merchandise site became obsolete.

Deadlines related to a competitor aren’t usually date specific but are focused on delivering enough functionality and excitement to make customers reconsider your product. Acme Media has followed this approach, and the first release of the Auctionator will stop the bleeding; subsequent releases will allow Acme to separate itself from the competition.

To Meet a Predetermined Schedule

Many companies release new products on a regularly scheduled basis. Greg’s current team releases a new product every 7½ weeks. The great thing about a regular release schedule is it gives you one solid reference point in a world of moving parts. The team acclimates to the deployment cycle, and the customer knows there is always another bus coming if a feature request doesn’t make it into the current release.

A fixed-release schedule also works well with a product backlog. You can see an example in Figure 21.1.

If you release products consistently, you’ll review the backlog consistently and develop a rhythm for evaluating features and scoping the forthcoming release with the customer.

Figure 21.1 Many teams follow a predetermined release schedule to provide rhythm for the team and the customer. A predetermined schedule lets you schedule iterations in advance and also removes customer anxiety when a feature isn’t completed in a given release. The customer knows the “bus schedule” and can get their feature on the next bus.

When There Is Enough Value

The preferred method for releasing any project is when your customer says there is enough value to deploy. Because customers are highly involved in Agile projects, they can say whether enough value exists at the end of each iteration.

Note

Why Not Release Every Iteration?

The potential complexity of releasing explains why teams seldom deploy each development iteration. Releasing each iteration can be expensive, and the cost of doing so needs to be justified. The expenses can include additional labor and blocking customers from accessing the system during deployment.

The Acme Media team didn’t deploy their code at the end of iteration 1. Iteration 1 provided enough core features to support a minimal system, but Jay, the customer advocate, wanted the team to spend two more weeks completing the secondary features so that a robust system could be deployed. From Jay’s perspective, they have only one shot to get their customers back from the competitors, and Jay wants the system to provide a solid user experience. The features provided in iteration 2 will support that goal.

Acme Media also understands the expense of deploying. Although many of Acme’s publishing systems support real-time deployment, the architecture of the Auctionator is more complicated and requires a unique deployment plan and additional mechanisms for operations and support.

Some Agile teams use charts to indicate how much value has been delivered. These charts are a great visual for the team and help support common understanding of status, but determining whether there is enough value to deploy is a subjective exercise. We frequently find that the customer makes the decision to deploy based on their perspective of status as much as what our tools indicate.

To Test the Product

In theory, you don’t deploy until the product is stable, but in some cases you have to deploy to validate stability. Sometimes you deploy to the production environment because you can’t simulate the final experience within your test environments.

For example, how many companies can afford to have a load balancer in their test environment? How well can a load simulator emulate a real user? How many PC configurations can you simulate in your test environment?

All the tests you do before you go live try to emulate what will happen when you deploy. But at the end of the day, there is no way to be 100 percent sure what will happen until you put the code out there.

In recent years, we’ve worked with many teams that understood this reality and designed soft launch plans to test the code in production. Note that these teams didn’t expose the functionality to all users but to a small subset who piloted the functionality and provided feedback.

Note

Making Sure a Constraint Is Real

Understanding constraints in detail often reveals that the constraint isn’t as demanding as you imagined. We’ll illustrate with a story from Greg.

Greg recently worked on an intranet application with his team. They were given a deadline for turning a new application over to their customer, and they began delivering iterations to prepare for deployment. As the deadline drew near, they identified a shortcoming in the design for load balancing. The issue would make it impossible to send email alerts to users. They revisited the design, came up with a workable solution, implemented the design, and began testing. Unfortunately, this last-second change cut into the team’s burn-in period. They wanted to run various performance tests with the system before deployment, and they estimated a month for this work. After the design change, they had only 2 weeks left.

The team spoke with the customer and found they had only one true deadline. Part of the functionality the team was delivering was to support a feature that was being lost when a vendor’s data center was shut down. If they could deliver that functionality as scheduled, they could continue to burn in and test the system and then deploy all features 2 weeks later than scheduled. That is exactly what they did.

Companies like Microsoft do this for every major product release. They offer the product to a beta group with limited support and noted risks, and then they review the feedback that comes in. Subsequent releases involve major customers and larger groups; then the product is released to the public at large.

Final Testing

Although every iteration is a releasable subset of the product, you still have to complete final testing before deployment. Potential areas for final testing are as follows:

• Functional testing of the last development iteration. You test code as it’s iteratively delivered during a development iteration, but you may still need time to complete functional testing of the final features.

• Final User Acceptance Testing (UAT). Some teams must go through a customer acceptance process before deployment. You may have to do this for regulatory reasons or because of the way the software development lifecycle is managed in your company.

• Final performance analysis. Many teams also do final performance testing before releasing. The last iteration allows us to test performance across all features and identify areas that may affect usability.

Before we look at each of these tests in detail, let’s discuss the relationship between quality level and the release decision.

What about Quality Level?

We’ve sat through many bug-review meetings where we discuss bug criticality level, potential workarounds, and how much the bug affects usability. We also estimate how much work is required to fix the bug. If a bug will take more than a few minutes of work, we label it a feature and then prioritize it with the other features in the current backlog.

In the early days of software projects, we had hard rules like “We won’t deploy with any level-3-severity bugs.” This was a nice rule, and we tried to follow it; but at every release, push came to shove, and we had to make the call about whether to miss our date or let some level-3 bugs go out with the deployment.

We’d love to tell you how Agile makes this decision easy, but even Agile has limited influence on the release decision. No matter what methodology you use, there will always be questions about whether quality is good enough to allow deployment. The good thing about Agile is we have baked in quality with each iteration so there should be fewer issues with the quality level.

Related to this question, we’re amazed that we can still find articles on the internet today that look at this quality-level question from a binary perspective. We recently read an article by a person with a significant development background, who said they would rather ship late than risk putting a buggy system into production. We appreciate this perspective and consider the thought noble, but we have trouble supporting it from a real-world perspective. We’ve worked at several startups, where the difference between shipping on time and delaying for improved quality can mean closing the company doors. Customers may go somewhere else, and investors may pull funds.

Conversely, we’ve worked on projects where the company wasn’t at risk, but we felt pressure to release early and put out a product so buggy that it hurt our reputation and affected future sales.

Agile does help with this dilemma by directly involving the customer in the release decision. You don’t have to guess what the customer would say; they tell you directly, and that has an impact on your decision. In cases where the customer is represented by a proxy, such as a product manager, the decision becomes more difficult. In these instances, the project team and stakeholders focus on understanding each other’s point of view and then reaching a consensus. On rare occasions, we’ve seen teams truly split on what to do; in those cases, a person such as the platform owner makes the call about whether to deploy.

Completing Functional/Usability Testing

Although you test iterations as you go, you may do slightly different testing before releasing. You may do tests outside of the team, such as usability testing and experimental testing. When you’re part of the team, you know the requirements, and you’re always looking to test against that vision. People outside the team haven’t been preprogrammed on what to expect and will find things you never thought of.

Usability testing involves bringing in members of your target audience and watching them use the application, frequently without instruction. The testers may be given light instruction of the ultimate goal, but you try to turn them loose to see if they use the system the way you intend them to. You do usability testing during iterations too, but that testing may be limited to prototypes and workflow simulation.

One of the interesting things about usability testing is you don’t always act on the discoveries. For example, if a user will use a feature multiple times per day, you may want to see how they adapt to a feature after using it several times.

Completing the User Acceptance Process

Although you’ve demonstrated the product to the customer during each iteration, it isn’t uncommon to have a final approval process prior to releasing. Common reasons for final acceptance testing are as follows:

• This is the first time the customer has been able to view the entire system.

• The project is tied to a contract that necessitates validation of requirements.

• The demonstrations to date have been with a customer advocate, and there is a desire to expose the system to a broader audience of end users.

After the customer verifies functional requirements, you need to make sure nonfunctional requirements are also supported.

Validation of Nonfunctional Requirements

Before you release, you need to make sure the underlying system performs as required. System requirements are frequently overlooked by the customer, and the project team can coach the customer in determining nonfunctional requirements.

System requirements don’t match up to a user story or specific functionality; instead, they represent the overarching ability to support service requirements. For example, Dairy Queen may have to support a user story where a customer orders and receives an ice cream cone. The user story steps may include customer orders the cone, employee creates the cone with the ice cream machine, and the employee delivers the cone. The user story meets the needs of one user, but it doesn’t say how many ice cream cones the system can support during a day. How many cones you must deliver in a day is an example of a nonfunctional requirement.

Nonfunctional requirements are as critical as the ability to support a specific user story. If the system fails, you won’t be able to support any user stories.

Experienced customers understand the importance of nonfunctional requirements, usually because they have paid the price for not specifying them in the past. These customers provide service-level requirements that are used to create a servicelevel agreement (SLA). The SLA helps the customer and the team reach agreement about realistic expectations of the system.

Here are some common areas to validate before going live:

• System availability. How much downtime is acceptable for system maintenance and system issues? If the system goes down, how long is acceptable? This number is usually specified as a percentage, and this percentage maps to the number of hours a system can be down in a given time period. Acme Media has a platform availability target of 99.5 percent. This means the company’s sites can be down for approximately 400 hours in a year.

  Note that increasing uptime percentage usually increases platform expense. For example, Acme Media feels it can support 99.5 percent uptime with its existing platform. If Acme wanted to pursue an uptime of 99.9 percent, the company would have to double the number of servers used and implement a complicated redundancy system. In effect, this would double the cost of the infrastructure.

• Data-recovery requirements. If the system goes down, how much data can you afford to lose? The metric for data recovery is Recovery Point Objective (RPO), which is specified as the number of minutes of data that can be lost during a failure. Acme Media determined that it can accept 10 minutes of data loss for the Auctionator.

• Downtime for a given failure. System availability relates to how much time you can be down for a given timeframe. Your Recovery Time Objective (RTO) specifies how long you can be down for an individual failure. Many companies have a second environment they can switch to in case of a failure. The RTO usually ties to how quickly you can switch to the Disaster Recovery (DR) environment. RTOs can range from a few minutes to a few hours, depending on the criticality of the system and how much money your company can invest in a recovery system and process.

• System response time. When a user requests a page, how long should it take for the system to respond? How long is beyond your usability threshold?

• Maximum concurrent users supported. What is the maximum number of users that can access the system at one time before performance begins to degrade.

• Archive and purging requirements. Over time, your system may become loaded with historical transactions and other data. How long does this data have to be available to the system? When you remove it from the system, do you have to store it for a period of time because of regulatory or other requirements? Acme Media doesn’t have a regulatory need to store the Auctionator transactions, but the company thinks it will be prudent to keep the log files for 1 year in case there are any questions.

In many cases, nonfunctional requirements are determined by the company providing the service (you) versus an individual customer. For example, Acme Media’s product manager identifies the nonfunctional requirements for Acme Media’s platform based on what he feels will be acceptable to customers.

Preparing Support Groups and Processes

Whether you use Agile or not, all projects need to have support processes in place before you go live. The main difference with Agile projects is you think about maintenance and support starting on day 1 of the project. Agile teams often use a template to record maintenance concerns along the way. Let’s look at an example.

The Running Maintenance and Support Worksheet

Acme Media labels its maintenance scratchpad the Maintenance and Support Worksheet. The Acme team noted their first maintenance concerns when the product was being examined for feasibility, and they recorded their final maintenance concerns during the last development iteration.

Some typical items recorded on a maintenance worksheet are as follows:

• Location of supporting documentation. Acme stores its support documentation on a network drive.

• Marketing information. This information may include a marketing plan or other information related to publicizing the application you’re about to release. Note that marketing a project goes beyond external customers. If you’re delivering an internally consumed application, you should still look for a way to advertise it to your users. This may include discussions at user group meetings or a notice/advertisement on your intranet.

• Known issues or limitations. What bugs or defects will be outstanding when you go live? Document these for support people and to make sure they’re included in your help documentation.

• Support for analytics. Many companies use an analytics tool such as Sage Analyst or WebTrends.

• Supporting jobs and processes. What jobs need to run nightly, weekly, or at other intervals to maintain the system? For example, the Auctionator requires a job to send out nightly email alerts to users who want to be notified only one time per day.

• Archiving. Production systems can become bloated with old data if they aren’t purged frequently. Acme Media establishes a nightly job to archive transactions that are 1 year or older.

• Search support. Many applications have a search engine tied to them. You may want to update the search engine for the application or feature you’re about to release.

Acme’s completed worksheet appears in Table 21.1.

Table 21.1 Maintenance concerns are recorded through all phases of a project via the maintenance worksheet.

Maintenance and Support Worksheet

Location for supporting documentation

Marketing URLs

Known issues and limitations

Redirects

Analytics tracking

Support jobs

Support and help articles

Archiving

Search indexing

Just like the other templates we’ve discussed in this book, you can use your worksheet as a starting point and then modify it to match the maintenance areas you need to track during a project.

Finalizing Help Materials and Support Processes

Acme Media treats its end-user help materials as a feature and delivers them in iteration 2. This is common with online web applications, where minimal user help is provided. In other environments, you may have a person or team dedicated to technical writing and the creation of support materials.

Acme’s help desk also has an escalation path so they can pass an issue to the development team if the issue can’t be addressed with their help documentation. If the issue isn’t severe, the help desk sends an email to the team. If the issue is severe, the help desk has a call tree and can work their way through it until they can find a person to work the issue.

Enabling System Monitoring and Creating an Escalation Process

Most companies monitor their networks to make sure the infrastructure is up and running. This includes monitoring database activity and server performance. In addition, many companies have tools in place to measure the user experience, including pageresponse time and application availability.

Acme Media uses a tool for measuring and monitoring the user experience. This tool, HP SiteScope, emulates a web browser and identifies potential issues before users are affected. Acme modifies the configuration of SiteScope so it will monitor various aspects of the Auctionator and trigger alerts to the team if potential issues are identified.

After you’ve established monitoring, you need to outline a process for managing alerts. Acme Media has a standard triage process for alerts so the correct people are notified if an alert is triggered by the Auctionator. Non-severe alerts go to the development team inbox. Critical alerts go to the development team and support team inboxes, and the support team calls development if they don’t receive a response within a specified period of time.

Enabling Maintenance and Background Processes

Acme Media notes two jobs that need to run to support the Auctionator: the first job archives auction transactions that are more than a year old, and the second sends out alerts to potential buyers on a daily and weekly basis.

The software projects you pursue may have many background process needs. In addition to the ones Acme Media identified, you may need support jobs for expiring content, data-warehouse population, reporting, refreshing content or data-search crawling/indexing, or pre-building cache on your servers.

Previous article: Adapting During Adapt Week

Continue on to the next article: Release: Final Testing and Support

Copyright

©2009 by Manning Publications Co. All rights reserved.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.