Overengineering overly simplistic solutions

I have noticed through working both at Microsoft and in other places a tendency for the two following practices to occur.  At first, it would seem they contradict each other but in truth I very often see them together.

1) The product is made to fit one kind of ideal user, most often the user who is the least tech savy.  While the team may use multiple personas, almost every feature must still be accessible and understandable to this user.  While this may help in some everyday programs, I find it particularly dangerous in APIs - where a class is designed for only simple uses and when confronted with a more advanced use case becomes completely useless.

2) The underlying mechanisms that make up the product are overly complicated.  In some ways, it's almost like the developers were bummed that the product itself isn't that complicated, so they intentionally make the code that runs it complicated.

Perhaps these are two different issues, but I suspect they are heavily intertwined.  For instance, it is unfortunately a common practice here at Microsoft to not use our own APIs to accomplish things.  While this has improved recently, the main reason developers give is the API doesn't do exactly what they want or doesn't perform as they need.  Here are some nightmare cases that I have actually seen (note: none of these occurred in my current group and to my knowledge this product no longer ships).

a) A development team did not trust the implementation of ATL, so they created their own version from the ground up.  Predictably, it suffered from numerous hard to debug issues.

b) A developer decided to learn more about multiple inheritance in C++, so he implemented the entire feature with it.  When the bugs mounted up, the developer left the team because he was sick of fixing them.  The feature was eventually removed from the product shortly before shipping due to instability.

c) To save a few MB of download, several developers decided to create their own image format to replace the PNG format.  The effort took several months, crippled the product until late in the design cycle on a legacy OS, caused the product to take 10% more memory, and resulted in a 5% performance degredation of the product.

These are plain examples of the code behind a product being unnecessarily complex.  Had the product team used ATL, avoided multiple inheritance, and stuck with the PNG format the code would have been shorter, customers would have seen fewer crashes and other bugs, and the product team would have finished sooner or have been able to add more useful features.  While these examples are extremes, I have seen the same types of issues occur many other times.

Very often, however, more advanced users of our products become very frustrated when they try to do something more difficult.  For example, recently I became very frustrated with the ServiceBase and ServiceInstaller classes that ship in .Net.  First, I needed to be able to install a service myself without the user getting involved.  This is possible, though rather tricky, to do with the ServiceInstaller class.  Eventually I wound up having to PInvoke all of the code because a number of things I needed to set on the service weren't available in ServiceInstaller.  ServiceBase soon suffered a similar fate, as I was playing around with shared services which ServiceBase had poor support for.

I won't delve much more into APIs though because it is truly very hard to get things right there.  From a UI point, I cannot count how many times I have needed to do something in Word and Excel that was hidden in some strange place.  Of course, that's when I was lucky enough to find it.

Note that this should not be construed as Microsoft bashing.  Every company I have ever worked for had an element like this.  I once worked for a now defunct dot com that was creating a web site for the customer.  The web site had to be as simplistic to use as possible - so users would need minimal training.  However the implementation consisted of the team trying to get several different products to work together.  I asked why they had such a strange colloboration of products and the answer was - "we wanted to learn more about them and also see if we can do it".  Unfortunately for them, the products didn't work and their solution never shipped.