Unhappy with Agile … revert to Waterfall. Is this really The answer?

Getting off the track

Clipart Illustration of a Stressed White Employee Staring At Stacks Of Paperwork On A Desk, Trying To Figure Out Where They Can Put Their Computer Keyboard I recently noticed a question on a distribution list that I have seen and experienced so many time before. A team tried the “Agile” community, failed and plans to move back to another methodology, such as the prescriptive CMMI or Waterfall, claiming that the processes are proven, reliable and will take them on the road of success. Personally, it may be years of experience in the trenches or hard learnt lessons learnt (I have a few project scars), I do not believe any of the defined methodologies will lead any team to certain victory.

Let’s take a huge step back and consider family X tidying up, dusting and vacuuming their family home. Family X gathers around the kitchen counter and plans the day to the nth degree, then proceeds to clean in a surgical and Swiss controller manner, by first tidying up each room, then dusting each room and finally vacuuming. Family Y gathers in the first room, agrees on the strategy for the day and then proceeds from room to room, tidying up, dusting and vacuuming … at times several rooms are being worked concurrently as not all three activities take the same amount of time. Family Z proceeds through the hours in no predefined sequence, giving the appearance of a chaotic bulls run through a town in Spain.

  • Question: Which family does a better job?
  • Answer: We do not know, but as we defined no special expectations, the end result probably looks the same … they all have a tidy and clean home.
  • So what? Each family used the process that suited them best to achieve the objective of cleaning their home. Over time we may see that family X and family Y will probably complete the cleaning exercise a lot quicker than family Z, learning from mistakes and challenges and adapting accordingly … that is, if they remain in the same home. If they change homes frequently, family Y will likely be the quickest to adopt to the new environment.

image … loaned from book “Software Engineers on their way to Pluto”, showing different project team environments

In information technology environments we often find that teams working like family Z, often believe that they are using an Agile approach … a misunderstanding that is causing great confusion, often frustration and leads to teams believing that doing it the old and trusted, prescriptive and meticulous way is a better approach. Wrong!

So, what should we be asking ourselves?

As mentioned in SDLC – Software Development Lifecycle … what’s the point? (part 1 of many) the questions teams should ask themselves:

  • Why are software projects always such a challenge to manage, deliver on time and as per specification?
  • Why are costs and efforts required high and why do these often increment exponentially towards the end of a project?
  • Why are our customers finding the error in the software for us?
  • Why are so many software engineers against a process framework, a SDLC and the tools that allow us to track, control and predict software ecosystems?
  • Do software engineers actually understand the value of a SDLC and what concepts such as test and build automation could do to their productivity, stress levels and home life?

At the end of the day there are no oracles in the world of Software Development Lifecycles (SDLC) and Application Lifecycle Management (ALM), there are no silver bullets and there is no methodology better than the other. What makes a methodology a success is the team than embraces it, using it as a supporting framework to promote stability, predictability, information sharing and using it as an aid, not as a solution.
image … loaned from book “Software Engineers on their way to Pluto” … view a methodology (hammer) as a tool, not a solution.

I personally know immensely successful teams working with CMMI, Waterfall, Agile, Scrum, Chaos and many other methodologies, or derivations thereof.

So, let’s briefly look at the basic project flow, then compare the agile and waterfall approach and conclude this long post by looking at my favourite methodology (framework) and two of my favourite projects and the approach we used.

Looking at the basic project flow


When we peel off all the layers of the various methodologies, we get to the core, the basic flow of any project. We typically start with communicating the problem domain, planning the vision and the scope … are we building a hobby plane or a space shuttle …, designing the deliverable, constructing, deploying the final product and eventually maintaining what we have. The better the quality of the deliverable, the cheaper, the more effective and the more enjoyable the maintenance. It is also important to stress, that the maintenance must be subject to the same vision and scope, the same quality and the same principles that the solution was built on.

Why you ask? Would you climb onto a modern Airbus A380 that is maintained by motorcycle mechanics in a backyard? I would not …

Asking ourselves what the methodologies are

What is Waterfall?

In SDLC – Software Development Lifecycle … exploring common models (part 3 of many) we note that Waterfall is known as the classic life cycle, whereby the process defines a sequential and linear approach to software development, including the associated communication, planning, modeling, construction, deployment and maintenance phases. This is how we wrote software in the 80’s, when it was very expensive and difficult to write code in Assembler, C and later C++ without a comprehensive design. We would often spend weeks to months designing a solution, before the coding would begin … a very linear, a very  prescriptive and often a very successful methodology.

image  … loaned from book “Software Engineers on their way to Pluto”, an organized, prescriptive and orderly environment

Personally I would recommend the more formal methodologies for systems that require a detailed design and a high degree of certainty before code is written, such as autopilot systems, medical systems and space probe systems that cannot be patched easily.

The latter, the space program, raises a few eye brows … while Pioneer is the furthest man-made object in space at a proud 91.865 AUs, Voyager 1 is the furthest still operational in space at 81AUs … that’s 12,116.6 million kilometers from Earth … and is expected to continue transmission of data until 2020 … that’s 42 years of operational time. Two very successful success stories … then again, we have space orbiters that recently crashed on Mars, because part of the team worked with the imperial and the other part with the metric system … millions in investment and unimaginable research lost because of a glitch that should have been detected within a very formal and prescriptive environment. Why such an error was not detected back on Earth, or more importantly why our planet has not standardized on one systems for speed, weight  and distance goes beyond my understanding … fortunately it is also way beyond the scope of this post :)

What is Agile

The Agile Manifesto states:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

While the principles are sound and the manifesto does say “while there is value in the items on the tight (non bold), we value the items on the left (bold) more”, it leaves it open for a lot of interpretation. Unfortunately I have seen teams that claim to be agile because they literally abandoned the items of the right (non bold) entirely … that was never the intention of Agile and leads to a chaotic environment, not an agile one.

In SDLC – Software Development Lifecycle … agility strikes back with energy (part 4 of many) we note that the objectives of Agile engineering seems to include:

  • Successful delivery of systems quickly
  • Customer satisfaction
  • Incremental and frequent software delivery, in the magnitude of weeks (preferred) to months
  • Collaboration between business and software engineers
  • Measurable and sustainable development
  • Quality, quality + quality solutions

image  … loaned from book “Software Engineers on their way to Pluto”, showing a dynamic and agile environment

Hang on, what is Chaos?

Many teams that believe they are using Agile, are in fact using a very chaos oriented methodology. developing a system without a design, keeping formal processes and specifications to a bare minimum or avoiding them completely, writing and evolving the solution with time, based on reactive scope creep or feature changes and many of the other common practices we often see in the trenches is not Agile, it is Chaos. In our Software Engineers on their way to Pluto book we refereed to them as the Churn team, because that is exactly how they normally operate. Oh, you need this feature? No problem, we will add it quickly. Normally these teams are very successful, until one day one of the key members decides to take a vacation or calls it a day … that’s normally when chaos sets in, expensive maintenance begins and re-writes become more cost effective than getting a new team skilled up on the solution and capable of supporting it. Does this ring any bells by any chance?

image  … loaned from book “Software Engineers on their way to Pluto”, showing a chaotic environment

Why do I personally prefer Agile (Scrum)?

I grew up and survived a lot of indoctrination of formal and waterfall methodologies and practices, but have grown to appreciate the flexibility of the agile principles. While I am personally not a fan of extreme programming, which falls under Agile, I like what Scrum has to offer. That is not because my years and years in South-Africa have injected a good portion of love for the game Rugby, which also has Scrums, but because I have personally experienced the value of collaboration, visibility of issues, success stories and challenges and the ability to react to change and still be predictable.

The tools and technologies of today allow us to prototype effectively, to develop code while creating documentation and designs concurrently, to automate a lot of the labour intensive and error-prone tasks, such as testing and building, and to be far more open to being reactive and responsive to change. Back in the 80’s I would probably not have embraced Agility and especially the Scrum framework, although we already used many of the concepts back in the 80’s and as part of more formal methodologies.

Today with the Team Foundation Server (TFS) and Visual Studio Team System (VSTS) we have an environment and a set of tools that supports any of the methodologies, especially the Agile ones. In SDLC – Software Development Lifecycle … is prototyping part of the good, the bad and/or the evil? (part 5 of many) we looked at prototyping, the pros and cons and after evaluating and working with TFS and VSTS 2010 (CTP and BETA to date) I personally believe that the time if Agility has arrived.

Clipart Illustration of a 3D White Person Leaning Against A Reflective Red Heart My favourite projects as an example

When people ask me about my favourite software project, two always come to mind.

The first involved a solution that required a concurrent load of 1,000,000 users, but only scaled to just under 20,000. We had to analyze and understand the solution, we had to recommend performance and scalability changes, we had to build a test harness capable of the huge concurrency test and associated volume and we had to take the solution to one of the Microsoft labs and test it. All this had to be done in less than 6 months … no time for formal processes, for formal documentation or the luxury of a linear timeline. We had to work in parallel … our team consisted primarily of one very agile rocket scientist who worked on the performance and scalability, one very Swiss analyst and developer who worked on the test harness, a very formal, precise and competent project manager and a number of developers and testers who joined re-enforced us on and off. I do not believe that any of us could have achieved the sheer impossible, but as a team made up of extreme opposites in terms of methodology and approach, we worked in a very Agile manner and eventually tested the system. We just missed the 1 million mark, not because the system could not handle the load, but because the lab literally ran out of hardware to host the million test bots. What this solution showed me is that a custom methodology and approach, made up of a variety of different brains on a stick, taking advantage of the best features of a variety of processes, could lead to success. We created an environment that suited us, ending off the recommended approach by the vendors and the customer.

The second project involved one that looked like one heading for a very chaotic death march. It was made up of a number of very bright and enthusiastic architects, analysts, developers and testers, but they seemed to get nowhere. We recommended to implement just two pillars of scrum, namely a daily stand-up meeting, backlog and sprint planning, with visual reporting … using very crude Excel spreadsheets. We also insisted on a daily build, because there was no consistent pulse visible and neither the developers or testers could give us a status report that created a warm and fuzzy feeling. Within a few weeks the team began to understand the value of the collaboration and was looking forward to the daily scrum. We were also getting flak if the daily report of the nightly build, the test run and the burn down chart was not visible when the team members walked past the coffee area in the morning, where the reports used to hang … visible and in the face of everyone … including the customer. The team which was initially made up of distinct pockets of business analysts, developers, testers, management and customer grew into one team. All of us sat through one very special night, when we had to implement one of the first releases … in complete darkness and a portable power generator keeping a skeleton number of systems operational during one of the regular blackouts. What was amazing to me was seeing people who were not needed that night, but sleeping all over the place to be close to the action, while others made sure that there was a constant supply of hot coffee and fresh pizzas … we had one team pursuing one team objective! Was it the Scrum methodology? No … but Agile principles brought tangible value to the team, the scope and especially the scope creep was managed by the team and most importantly, the team embraced parts of the methodology to its benefit and its ecosystem. Had we enforced any SDLC/ALM methodology on the team, we would definitely have failed.

To conclude a long story … so what is the right methodology?

It is like the answer 42, it depends. Instead of trying to find the answer to this question, explore the available methodologies, pick one that matches “your” team and environment, adapt it to fit in with “you” not vice versa and evolve over time. If the one methodology does not work for you, ask yourself why. Is it the methodology or the way you implemented it? First determine the “why”, because simply switching to another methodology could mean that you are lining up the team for another culture change and another possible failure. It really does not matter whether you use CMMI. Waterfall, Incremental, RAD, evolutionary, component based, aspect based or a very formal methodology. What counts is the end result and your team. A happy team, enjoys what they do and will deliver quality solutions again and again.

My dream would be to write a practical SDLC/ALM book, based on today’s tools, for all of the stakeholders of a typical project, ranging from the sponsor, the project manager, the developer, the tester, the maintenance engineer, the technical lead, the architect … everyone. Unfortunately it will be difficult to design a book that is appealing to all and a publisher who is willing to take the risk of not publishing another academic book.

But, we can always dream :)