Breaking the cycle of failed development projects

It's been a bit over two weeks since I arrived back in Australia, and things are slowly coming together. We're still waiting for all of our stuff to arrive, and for that matter a place to put it all - but at least it's sunny every day, and I'm enjoying my new role at Microsoft's Solutions Development Centre.

While it is strange going back to the world of consulting, I'm sure my tenure with the patterns & practices team will give me some new perspectives for the role. As an example, the subtle persuasion of people like Peter Provost has made me an agile development convert (although certainly not an expert!). While I believe there is a growing trend towards more agile approaches in many organisations, the p&p team is definitely ahead of the curve - but I'm very interested in applying what I've learned about agile development in my new role.

But I suspect it will be an uphill battle in many cases. One of the first tasks that was thrown at me was to read through a draft "request for tender" document from a customer. This document was essentially asking for a fixed scope, within a fixed delivery schedule, for a fixed price. I realise there is nothing particularly unusual about this request. But we all (hopefully) know that it can't be done, short of the odd fluke. Each project is governed by the "iron triangle" of scope, schedule and resources. You can choose which ones are fixed and which are variable, but you can't fix all three without sacrificing quality, unless you are extremely lucky and everything comes together within the expected parameters. Depending on which reports you believe (such as the Standish CHAOS reports), only 15-30% of projects finish on schedule and on budget, a similar proportion fail outright, and the remainder fall somewhere in between with the project delivered but not meeting expectations.

The thing I am trying to understand is why we (meaning both the development teams and the business groups) insist on playing this game by pretending it can all be done. Business, of course, wants the certainty on what they will get when and for how much. Most development and consulting shops will take a project with a relatively detailed (but never final) requirements list and fixed schedule, and happily provide a fixed price quote (probably with a hefty risk premium attached). But with such a poor project success rate, surely everybody should know that the emperor has no clothes? My cynical explanation is that this model makes it very easy for everybody to blame someone else. The business can blame the development teams for failing to meet the agreed deadlines or requirements. The development teams can blame the business that the initial requirements were not detailed or accurate enough. Everyone walks away vindicated that it wasn't their fault that the project failed.

This is where agile development approaches should really be able to help. I've had no formal training on agile so forgive me if I misrepresent anything, but as I see it one of the most fundamental concepts is that scope is never fixed, but is aggressively and continuously prioritised. Nobody should promise exactly what will be delivered at the end of the project, although it makes sense to do some upfront analysis on the minimum and preferred scope to get a reasonable idea of the how long it will take and whether this will fit into the proposed budget. Another key agile concept is the project will be broken into multiple iterations and releases, each of which will be deployable and provide real business value. This approach allows requirements questions to be addressed continuously based on customer feedback, allows scope to be reprioritised throughout the project, and mitigates "big bang" releases that often result in the business saying "this is just what we asked for, but not what we want". Finally, in theory at least, a well-run agile project should never ever run over time or over budget. It may not deliver all of the functionality the business wanted, but the functionality that was delivered should at least be usable and valuable, and more important than any scope that was not delivered. 

Anyway, enough of Agile 101. My real question is more about psychology than methodology. What will it take to get people to understand, and act on the fact, that the way most projects are run doesn't work? Most of the agile success stories I hear about (including p&p, it turns out) have been grass-roots campaigns from developers who slowly convinced management teams in their own organisation to move to more agile approaches. This is a great start, and I guess it's gotten us to where we are now. But I'm not personally familiar with any examples of an external systems integrator or development shop successfully convincing a non-agile customer to think about their development approaches differently. Things get even harder in a tender situation, where the customer will have set ideas on how tenderers should respond to allow for an "apples to apples" comparison. If one response says "yes, we will deliver all of the scope in X months for $Y", and another says "We will deliver something in X months for $Y, but we can't tell you what it will be yet", you can bet the customer will choose the first option, even if the second has a far better chance of success.

I'd love to hear your thoughts on all of this, especially since I've been living in an alternate reality away from the consulting world for quite a while. Is the general state of software development really as bad as I make out? Is agile development a critical factor in addressing these problems? Have you been successful in introducing agile approaches in your own organisations or to external organisations? Is it possible to win a tender by proposing an agile approach when the customer wasn't expecting it? What do you think it will take to break the current cycle of blame and get everyone to think differently about software development?