Tragile Documentation

I just discovered last week that I'm supposed to be able to "move quickly and lightly", be "as sleek and agile as a gymnast", and be "fleet of foot". Either that or I'm supposed to be an X-ray and Gamma ray astronomical satellite belonging to the Italian Space Agency. Not much hope of any of these happening, I guess. Probably I shouldn't have decided to search the Web and see what "agile" actually means (and, in case you are wondering, the Italian satellite is called Astrorivelatore Gamma ad Immagini LEggero - see Carlo Gavazzi Space).

All this comes about because one of the projects I'm on at the moment is effectively exploring the techniques for agile document creation. Agile is well-known in the software development arena, and it seems - from what I've seen here at p&p - to work well. OK, so it has its downsides from the documentation engineer's perspective (as I know only too well), but it does appear to produce great results very quickly, with fewer "issues" than the waterfall approach.

So, can documentation be agile as well? A lot of the supposedly related stuff on the Web seems to be concerned mainly with creating dynamic content for Web sites, or creating content that you can display on different devices (such as mobile clients). However, there are a couple of people talking seriously about the issues. You'll find great articles from Scott Ambler (see Agile/Lean Documentation and Can Documentation Be Agile?) and Ron Jeffries (see Where's the Spec, the Big Picture, the Design?).

Much of the focus appears to be on documentation in terms of the requirements of development teams (project documentation, such as requirements and design decisions) and operations (dependencies, system interactions, and troubleshooting). However, they do talk a bit about "customer" or "user" documentation. What's worrying is that all of the articles seem to recommend that writing user documentation should wait until the product stabilizes, be delayed as long as possible, and be written as close as possible to the product's release. Working on end-user documentation early in the project is, they suggest, just a waste of resources.

Yet, if you don't start early and "run with" an agile dev team, how can you hope to have comprehensive user documentation available for the release? In theory, the team will capture all of the information required in models, sketches, and user stories that can be polished up to provide the user documentation. But is that enough? Should the whole design and structure of the documentation be based on developer's conversations and implementation specifications? Telling users that it "works like this" and "can do this" does not always produce useful guidance.

For example, how many times have you opened up the help file for some complex server product and seen sections titled "Configuration", "Administration", and "Reference"? Maybe you want to do something simple, such as add a new account. You find the section in the administration topic called "Adding a new account" and it explains which buttons to click, which textboxes you need to fill in, and how to select an option for the "operations category". But it doesn't tell you why you need to do this stuff. Or what will happen if you do (or don't) select an operations category. What the heck is an "operations category" anyway? Maybe they are a fundamental new feature that got added to the product right at the end of the dev cycle, after the docs for creating a new account were completed. And so there wasn't time to do the fundamental reorganization of the content required to match the new conceptual approach you now need to take when creating an account?

Agile development, they say, means that "working software is more important than comprehensive documentation". But all the articles and references seem to end up talking generally about documentation when what they really mean is "project documentation" - stuff for the team and project managers to use to monitor and manage the project. You continually get the impression that customer-focused documentation is neither a requirement, or important. Yet, without it, isn't there a chance that nobody will actually use the software? Or will never get the maximum benefit from it?

Getting back to the question of whether you actually can do agile documentation for a software project, how might agile work? For a developer, a task might be something like rewriting an interception algorithm to improve performance, or changing part of the UI to make it easier to use. For a writer, a task is usually to produce a document that describes some feature. If that feature is sufficiently compartmentalized, such as describing how to use some part of the UI or explaining how to write some plug-in component, then it is an ideal candidate for agile. You can focus on just that task, and then move on to the next one.

You can pair with another writer, or - even better - pair with the developer to help you understand the feature, and then use your "professional writer's abilities" (hopefully you have some) to get it down in a form that makes sense to users. And, because you don't have the same internal insight into the feature as the developer does, you can ask "user-level" questions and document the answers. Where it gets hard is when you are trying to grasp the "whole concept" view, especially from dev teams that know the code base inside out, instinctively understand the overall concepts of the product, and can't figure why, for example, anybody would find it hard to understand what an "operations category" is.

I guess lots of people have carried out scientific investigations into all this, but my simple and humble take is:

• Prioritization. If everything is gradual iterations towards the end goal, but the end goal is not clearly defined at the start, how do you plan the documentation requirements and prioritize the stages so you work on the right stuff at the right time? How do you know if what seems important today will no longer be important in a week's time, and might even disappear altogether in a month's time? To be effective when creating documentation, you need to be able to base it on an overall plan that has definable stages.
• Concentration. Documentation development requires a similar approach, in terms of planning the architecture and linking the parts together, as code does. It also requires concentration, and is not a task that you can easily accomplish when paired, when time-limited, and when constantly shifting focus. Its fine for collating facts and information, but organizing and writing is still, I reckon, a "one-person" and "in a quiet room" task.
• Flexibility. Trying to flit from one task to another is tough enough for developers, but it's often much more difficult for the documentation engineer where changes to the product are occurring all the time. There are no tools that allow you to automatically find and update references and content elsewhere in a documentation set.
• Churn. While developers search for better ways to decouple their code components, in documentation you inevitably need close coupling. Topics can't link to some decoupled interface class - they have to link to a specific topic. Move, change, or delete the topic and it all breaks. Even a minor change to the functionality of a component in a large project can have a huge impact on the documentation. One way to tell a writer is to see if the labels have worn off the Page Up and Page Down keys on their keyboard.
• Feedback. Inevitably the writer does not have the same level of insight into the product as the people who are writing the code, and in most cases doesn't have the same level of programming skill (if we did, would we be writers?). And, in most cases, not having the same insight is good because it means that the writer can act as the customer's advocate. But it does require constant feedback and input from the dev team, throughout the project if the documentation is being developed alongside the product.

However, while these issues tend to arise when documenting a piece of physical software under development, the whole approach is different for the kind of task I'm currently engaged in. It involves producing guidance for technologies that are already released and are (generally) mainstream; for example, architectural guides and documentation that help you to use existing products. This is where you can actually work with something that is stable and finished (although they'll probably release an updated version just after you publish the guidance). Maybe agile works better here?

What's interesting, and initially prompted this rant, is that I'm discovering how the traditional (i.e. not quite so agile) approach still seems to have some fundamental advantages. Documents I create from scratch to achieve a specific aim, working on my own in a quiet place, seem to come out better that those that get passed around from one person to another with everyone "adding their bit" but not seeing the whole picture. Being able to prioritize work myself means that I can spend time on the important tasks without being worried that there is some more urgent task waiting. And seeing the whole picture means I can write with more accuracy and less duplication of effort.

I guess, in the end, I'm still struggling to see how you can use agile methods for end-user or consumer documentation. Maybe it needs to be some custom blend of techniques, concentrating on the features of agile and traditional methods that work best. Use the agile features of pairing for collating information, flexibility for maximizing effort, and feedback to keep it on target. Combine this with traditional approaches to prioritization, concentration, and an overall plan. Maybe it's a whole new paradigm? Can I trade-mark "tragile documentation"?