The Origin of LINQ to SQL

LINQ to SQL, possibly Microsoft’s first ORM to actually ship in ten years of trying, was never even supposed to exist.

It started out as a humble Visual Studio project on my desktop machine way back in the fall of 2003, long before anyone heard about it, long before anyone even guessed what would come next, except for the readers of this blog, of course, since I used to post often with long obtuse and sometimes psychedelic meanderings that with the proper one-time pad to decrypt it you might have possibly guessed what was on my mind. I needed that project to prepare for what was coming. I needed expression trees and a basic SQL translator to get ready for the day when I would get my hands on the C# codebase and turn it into language that could actually do something interesting for a change; a language that would know how to query. [Cue the FoxPro fans]

Luckily, it didn’t take me long to get the basics up and running. You see, it wasn’t the first time I’d slapped together an ORM or modified a language to add query capabilities; having already designed ObjectSpaces and parts of C-Omega so I was certainly up to the task. Fortunately, it gets a lot easier the ‘next’ time you design and build something, especially if it was you that did the designing before and you have the opportunity to start over fresh. I knew all the parts that I needed and which of those needed improvement from the last time around. So you might say that LINQ to SQL started out as ObjectSpaces vNext, or vNextNext, since I had actually built three different working prototypes of ObjectSpaces before settling on the one we went with, and then another simpler one underlying C-Omega.

Yet, I was not really setting out to create a whole full-blown ORM system. I just needed something to kick the tires of LINQ with, a ‘mock’ LINQ provider if you will. We all assumed at the time that eventually ObjectSpaces and even WinFS would adopt the patterns that we were devising. So I could simply have just started with ObjectSpaces, and to tell you the truth I wanted to, but I also knew that LINQ would need something a little different. You see, ObjectSpaces was originally targeted to come out long before even Whidbey; its original target was Everett (.Net 1.1). That might not sound too crazy if you realize that Luca and I started the project a few years before even .Net 1.0 shipped. (Back then my manager asked me point blank what I thought came after ObjectSpaces; what was the next big thing? It only took me a moment to realize that integration into a programming language was it, getting the queries out of the strings, getting compile time type-checking back and intellisense and all that. Not that I’m all that brilliant or foresighted, we had been already working on fleshing out ideas for new database programming language so it was pretty much on my mind before I walked into his office.) The big drawbacks of ObjectSpaces were its limited query capabilities (predicates only), its focus on queries in strings and the lack of strong-typing through generics. Even with ObjectSpaces delayed and retargeted around Whidbey, they were already too far gone in their design to start thinking about drastic changes, and besides, whatever I was doing to the language wouldn’t affect them until long after they shipped; they’d get around to this ‘LINQ’ thing in the next version.

So what I needed to feed LINQ was a language-integrated ObjectSpaces redesigned around generics and IEnumerables that processed relationally complete queries foremost via strongly-typed query trees generated by the compiler. That’s what I'd hoped ObjectSpaces v2 would turn into eventually, but I needed something now to design and build LINQ against.

Why didn’t I start with WinFS? After all, it was all the rage inside the SQL Server org at the time. Unfortunately, it was the same story as with ObjectSpaces. They were shipping before us. We weren’t on their radar. Their hubris was bigger than ours. Not to mention my belief that WinFS was the biggest fiasco I’d ever bore witness to, but that’s another story.

Yet, part of that story was the impetus to turn LINQ to SQL into an actual product.

The WinFS client API even started out as a complete copy of the ObjectSpaces codebase and had all the same limitations. It just had more political clout as it was being lead by a figure at a higher-point in the corporate org chart, and so it was positioned as part of a juggernaut that was making a massive internal land grab. We on the outside used to refer to WinFS as the black hole, forever growing, sucking up all available resources, letting nothing escape and in the end producing nothing except possibly a gateway to an alternate reality. Many of our friends and co-workers had already been sucked in, and the weekly reports and horror stories were not for the weak-of-heart. It eventually sucked up ObjectSpaces too and in the process killing it off so that in WinFS v2 it could all be ‘aligned’.

At that point, those of us designing LINQ got a bit worried. There were not too many in the developer division that believed in the mission of WinFS. As a developer tool for the masses, something simple that targeted the lower end was paramount. ObjectSpaces had been it, and now it was gone. There was still some glimmer of possibility that WinFS v2 might eventually get it right and be useful as a general ORM tool. But all hope of that was shot when WinFS was pulled out of Vista and its entire existence was put in doubt. Had they immediately turned around and brought back ObjectSpaces, that might have worked, but in the intervening months ObjectSpaces had slipped past the point of no return for being part of .Net 2.0, turnover within the SQL org was spiraling out of control, and most of the brain-trust that knew anything about ORM had already fled.

That’s when we realized we had no choice. If LINQ was to succeed it needed some legs to stand on. The ‘mock’ ORM I was building was shaping up to be a pretty good contender. We had co-designed it in tandem with LINQ as part of the C# 3.0 design group and it really was a full-fledged implementation; we just never thought it was actually going to be a shipping product. It was simply meant to act as a stand-in for products that now no longer existed. So, for the sake of LINQ and the customer in general, we took up the ORM torch officially, announcing our intention internally and starting the political nightmare that became my life for the next three years.