IQueryable’s Deep Dark Secret

I love the IQueryable interface, but it’s got a dark checkered past that most of you might not know about. IQueryable is a great way to expose your API or domain model for querying or provide a specialized query processor that can be used directly by LINQ. It defines the pattern for you to gather-up a user’s query and present it to your processing engine as a single expression tree that you can either transform or interpret. It’s the way LINQ becomes ‘integrated’ for many LINQ to XXX products. Yet it was not supposed to be that way; with all that ease of use, plugging automatically into LINQ with an abundance of pre-written query operators at your disposal. You were not supposed to use it for your own ends. It was not meant for you at all.

It was meant for LINQ to SQL. Period. The interface, the ‘Queryable’ query operators and whatnot were all part of the LINQ to SQL product and namespace. The original plan of record was to require all LINQ implementations to define their own query operators that abided by the standard query operator pattern. You’d have to cook up your own clever way to connect calls to your ‘Where’ method back to some encoding that your engine could understand. It would be daunting work to be sure, but not impossible. After all, you were likely building a query processor anyway, so what’s another thousand lines of code?

Of course, that was until that fateful day in December 2005 when the gauntlet was thrown down and the challenge was made; a challenge that had nothing what so ever to do with IQueryable. It started out as a simple little request by the infamous Don Box. “Why don’t you guys have an eval for these expression trees?” He said to me that day in an email. He was working on another project unrelated to LINQ and saw potential use of the LINQ expression trees for his own purpose, as long as there was some way to actually execute or interpret them. Of course, he wanted us to take it on. Yet our product was already so full of features that we were having a hard time as it was to convince management that even an ultra slimmed down LINQ to SQL would fit into the Orcas schedule. So I mailed him back with, “Yes, it should be straightforward to convert these trees into IL using the reflection emit API. Why don’t you build it?”

You see, I challenged him to write the code, not the other way around. I figured that would shut him up. Yet, to my surprise he actually agreed. He was going to do it over the holiday break, hand it over to me when he was done and I’d find some way to get it into the product. As it turns out, I was actually relieved. It wasn’t like we had not already thought about it. Most of the design team wanted there to be an eval like mechanism, but it was not high priority since the primary consumer (LINQ to SQL and other ORM’s) where not going to need it. So over the holiday break I actually built up anticipation for it. I was pre-geeking-out. What was he going to build? Did this guy even know how to write code? Would he figure out how to solve the closure mess? My god, what had I started?

As it turns out, Don did not find the time to build anything, and I was somewhat let down. However, I had gotten myself so juiced up about the idea of it working that I didn’t care. It just gave me the excuse to do it myself, and I love to write brand-new geek’n-out code. So the next weekend in January I spent all the brownie points I had built up over the break by engaging in ‘family time’ and plugged myself to my machine for an all night coding session. I was running high on adrenaline, and the solutions just seemed to come as fast as I could type. By Sunday it was all working beautifully. On Monday I was eager to show it off and so I did during the design meeting. I showed everyone a mechanism that could turn any LINQ expression tree into a delegate that could be called directly at runtime. The IL generated was the same as what the compiler would give you, so it performed just a well.

Of course, that’s when it happened. That’s when this seemingly unrelated geek-fest over the expression tree blossomed into something much more. You see Anders had been thinking about something else over the break. He was looking for a way to solve the polymorphism problem of making queries first class things within the language, since what we had so far was a really just an illusion. Query objects were really just IEnumerables. LINQ to SQL queries were IQueryables, which were IEnumerables by inheritance. The only way someone could write a general piece of code to operate over ‘any’ query was to specify its type as IEnumerable. Yet, the compiler would treat the query differently depending on its static type. LINQ to SQL’s IQueryable mechanism wouldn’t work if the compiler thought it was IEnumerable, no expression tree would be built; the query would just run locally and not inside the database server where it belonged.

After seeing the demonstration everything just clicked. If IEnumerables could be turned into IQueryables such that the operations applied to it were captured as expression trees (as LINQ to SQL was already doing) and if those expression trees could be turned back into executable IL as delegates (which so happened to be just what we needed to feed into the locally executing standard query operators) then we could easily turn IQueryables back into locally executing IEnumerables. The IQueryable interface could become the polymorphic query interface instead of IEnumerable. Queries meant to run against local objects could be manipulated just like their expression tree toting brethren. Dynamic mini-languages could be written to generate expression trees and apply query operators to any type of query generically. Life was good.

The whole was suddenly greater than the sum of its parts. It became obvious that we needed the expression compiler as part of the product and that IQueryable should be promoted out of the private domain of LINQ to SQL and into the limelight to become the general definition of a query. It was a done deal.

And all because Don wanted us to do Evil Eval.