Importance of engineering models before the actual build

Call me a control freak, but I really like to know what I'm going to get before I buy something. Not just how the box and wrapping looks, but the internal engineering details.

I used to work for CSIRO - with a group of other diehard computer engineers, we benchmarked, evaluated middleware technologies (Tuxedo, Encina, CORBA, J2EE, COM+/.NET, Message Brokers, BPM technologies etc) just for the fun of it! In fact, I think some of the reports are still here. Hey that was great! We then discovered that while we were having fun stressing these complex technologies, probing their internal architecture and workings, lots of private industries and Government agencies came to us to seek guidance and advice when it came to middleware acquisition decisions. And that was even more fun! Especially when you are able to get an understanding of their likely usage pattern, use cases, workload profile, and then show them via our benchmark environment what sort of behaviour they're likely to see once they deploy the middleware in their environment (in terms of performance, scalability, reliability etc).

It is quite sad that not all software engineering projects (where COTS middleware is involved) can afford the time and the resource to do this thorough evaluation beforehand. I personally think we provided an extremely valuable service at CSIRO, but unfortunatley our group kind of dissipated over time... it's now quite sad to see companies making the 'wrong acquisition decisions' because they don't have the information about 'expected quality' of the middleware. The industry is littered with these sort of expensive mistakes.

Perhaps next time I meet another respected Analyst from Gartner, I should put forward the proposal - that Gartner should expand their services to include 'rigorous middleware technology benchmarking and technical architecture evaluations', in addition to the 'customer interview based research' that they're famous for. I think lots of people in the industry would really benefit from that sort of insight based on a 'hands on engineering reconstruction' approach to software evaluation.