Software Automation and Enlightenment

I just left Microsoft because I hit an uncompromising roadblock in the company, in the specific group which I worked in. It was disappointing of course and liberating at the same time, but on reflection I can’t help feeling that the company is not really doing much to lead or help the software industry in standardization of software solutions or in automating custom software development. My work in that space for Microsoft, much of it hidden, has now concluded there, but before leaving I was fortunate enough to be able to release a major milestone in our work to the public from the secrecy and clutches of Microsoft internal, and transfer that ownership to the Outercurve Foundation, where the VSPAT project now has a chance to flourish out in the open. The primary objective of that move to Outercurve being: to allow others in the community (outside of Microsoft) to ‘see’ and benefit from its progress, contribute to its future, and more importantly to be able to start talking about and demonstrate its breakthroughs openly. Sadly, there is no group in Microsoft who is thinking about this set of features or these scenarios anymore. Most of the knowledge and expertise by those deep in this space has been disbanded or has left the company, or is focused on other priorities. But why is a tool maker like Microsoft not moving in this direction and demonstrating leadership in this area of software production? Why does Software Automation remain largely suppressed/unrecognized by the large tool vendors as a powerful driving force in moving the software industry forward out of the dark ages of a cottage industry?

Before I get too far on this perhaps controversial topic, the kind of automation I am referring to is not your typical refactoring, renaming or spell checking type of automation, but something much larger and structured, and that will inevitably contain all of that kind of automation too. The kind of automation that gives software communities/companies the power to automate parts of their own software development solutions, their own way, by giving them tools that build them custom tools for them that automate their own custom development the way they have learned to develop software. Those organizations are already doing software development their own way there is no doubt about that, but today it is sooooo tediously and painstakingly manual, error prone, and expensive with such inconsistency in quality and supportability. Creativity and Experimentation seem to always win over Productivity and Standardization in almost all cases, if you ask an engineer to decide. The kind of tools I am taking about encapsulate knowledge and expertise from members of the existing software organization that automate their own software development processes within their current tooling environments. The kind of tools that are built to integrate seamlessly with one another specifically for the project, and that can be easily customized in their native form to suit a specific project (i.e. without access to the original source of the tool). I am not talking about the various proprietary tooling and automation that companies like Microsoft release that helps you rapidly apply their latest technologies the way they might suggest you do it (think: the ASP.NET MVC type tooling they provide). I am talking about “tools that build custom tools for developing software the way you want it developed – your way!

The fun part about this topic is that not many people have either seen or have envisioned that such tools could possibly exist yet – hence the dark ages. So it is quite hard to find examples of where you have seen this kind of tooling in the past.

There are plenty of cool tools/extensions out there that do code generation or apply a specific architecture or a specific set of coding rules or coding patterns, technologies and practices (pick any ASP.NET MVC toolset, or Windows Phone development toolset, even the few Software Factory type toolkits out there etc.). These tools can be great at kick-starting development on a project, but invariably you will only realize their full value if you subscribe wholeheartedly to all the practices and patterns they may generate for you for those specific technology combinations they may have preselected for you. AND IF they fully address all the requirements of your specific project, AND IF only the components they use, the dependencies they take and the code they generate painlessly integrates into your own custom code in predictable ways that you understand and can control. Generally speaking, this succeeds only for so few projects, or for only so long in the project before the tools are abandoned or their outputs ripped-out and evolved manually. I’d say, in a great deal of cases that happens not much farther than the first demo or initial iteration with the tools. It’s not long before developers want to code around or abandon those tools, regain full control of the code and write/rewrite their code by hand again. Renter – rework, avoidable manual errors, code reviews, governance, distrust and loss of productivity again.

What every development team may embrace however (if they could get them) are their own customized tools, that bake in their own best practices, their own coding patterns, their own architectural constraints, assumptions, their own requirements for supportability. And to have those tools automate and generate the relevant code/components into their projects consistently in an agreed and predictable way that works for the members of their teams, in their organizational structures, on their projects. Done in such a way that it complements and supplements what they have learned and know to be best for their projects. They want tools that create the software their way, using their knowledge, their style, their rules and for their people on their projects, their way – period.

So isn’t providing that kind of power to them important progress to make? so that the industry may as a whole start to make progress on standardizing software technologies, and patterns, to bring down the cost of software development, and start reusing (within your own organizations at least) what is known to be proven to work in many cases for the same kind of solutions that you build? Think what automation, robots, and computers did for other engineering/production industries, and the products they build? The parts that can be automated are automated and made affordable, and the parts that needed a human touch are hand crafted to cost a hell of a lot more money to produce and verify!

It’s now almost 2013, and modeling and automation of software is still such a fringe part of the industry. Attend one of the modeling conferences and see for yourself – it’s not going anywhere very fast. It still feels like we are up against an industry that feels it needs to continue to handcraft every line of code just so, because all the automation and generation options out there require such an ominous commitment to some esoteric set of tools or frameworks that are so alien to what we already know in our work. And how to integrate with them? Just another quirky, unsupported tool or approach I don’t want to learn or take a dependency on in this project. The contributors of these tools/extensions will demonstrate and tell you that you can generate all the code you need for your web app just by pressing a few buttons, and up pops a web page prepopulated with data from a new database schema with all the stored procedures, tables, queries, data access layers and services you need without writing a single line of code – bla, bla, bla. This is a great start, but what then? Now, you are going to need to integrate those 3rd party controls or scripts you just purchased or built on the last project, you need to add custom logging and tracing code differently at each layer in your architecture, and ensure that everyone on the team (some of whom have never used this technology before) are writing the code the same way you do it. Then enter the necessarily hand crafted code that needs to be written from scratch because it’s unique to your custom requirements/logic/etc. How do you accommodate all of that into your new project and make it work with these new fancy tools you just spent weeks investigating and experimenting with? Mightn’t you just as well drop those fancy and unfamiliar tools and start from scratch and do it the way you know to do it from the start? then at least you can get it right, just the way you always did it? Don’t you think? Let’s just code it!

But maybe, perhaps maybe, you are ‘throwing the baby out with the bathwater’, no? It’s not that the automation and tooling stinks, or the code it may produce is rubbish to you. More than likely that generated code is far better optimized and efficient than the code being written by hand on the project anyway. And the components being put together in its architectural frameworks are probably far better optimized, and undoubtedly have multiple integration points for extending with custom code. That’s if your architecture has any such structure or extensibility at all! Rarely any of that provided value is bad for you – it’s usually quite good from many of the good automation toolsets. But I think the proposition of using those tools rarely ever works out well for so many development organizations because it comes down to a number of insurmountable human issues, that almost always include some combination of these understandable excuses:

1) The generated code from the tool was not written by anyone we trust on our team on this project, or we don’t like the way the code looks (i.e. its coding standards or style), and we don’t want to be bothered with learning how to change it. Therefore, we will have to spend far too much time and money quantifying and qualifying it, and *bingo* we won’t be able to support that code.

2) We don’t understand the architectural or structural patterns of the software that are presumed by this tool because they were created by someone we don’t know or trust for a set of assumptions we don’t understand, and for a far-too-general case than ours (perceiving that our project is far too unique, than anyone else’s). So, *bingo* we won’t be able to figure out how to integrate all our other custom stuff we are going to need in the project into all that crap to be efficient or elegant.

3) At some point we are going to want this tooling to do something special for us, and it probably won’t be able to and *bingo* we just won’t be able to cannot afford the time to learn someone else’s toolset right now to adapt it for ourselves. And:

4) Let’s face it, automation is boring and not creative – **let’s just code it all ourselves, experimenting is too much fun**!

We are still down to the big two human issues: trust and institutionalized knowledge. (That is: unique knowledge about how we do things our way, based upon what we know and our experiences of what works best here for us, on our projects in this organization/community). The two barriers that any proposed tooling need to surmount before being adopted widely by any software development team in their projects.

So how do you start to produce tools of this caliber that ever get over the hurdles above, and that could be welcomed onto a project to increase productivity and automate code development? Isn’t the answer: tools and automation that were purpose-built by the members of the present project team themselves? (or by trusted colleagues of theirs/authorities to them)? If they were to build the tools, then those tools will certainly do things in predictable ways that they know and trust, with their institutionalized knowledge baked in. Tools that create code and patterns that they are intimately familiar with. Tools that evolve as their knowledge and expertise evolves. Tools can be extended or customized at any time to add that extra little bit of new functionality later in the project, and *bingo* those folks may even start to encode in those tools a little standardization about how parts of their software are written in this (and the next) project by their teams/communities?

If you could get your hands on such tools, and can learn how to use them quickly enough, and get them to reproduce what you already know to be proven, then surely that is the first step in progress in Software Automation in a cottage industry? Try a powered tool for that job for quicker and better results!

Perhaps if enough organizations/communities demonstrate the overwhelming advantages they gain from this ability to automate their stuff, then others might have to find more ways to automate to stay competitive?

Who knows, someone might even start to suggest standardizing their own standardized practices and patterns and golly! perhaps we might get engineers to agree on standard ways of solving standard problems, and customers may finally start to get more robust and predictable software solutions at more affordable prices. And the industry itself could start to emerge into an age of enlightenment!