March 2009
Volume 24 Number 03
The Polyglot Programmer - Mixing And Matching Languages
By Ted Neward | March 2009
Contents
Polyglot Programming
Polyglot in Practice
The Cost behind the Choices
Long before the Web and client-server programming, it was common for an entire application to be written in a single language on top of a single platform or system. Consider, for example, the ubiquitous FoxPro platform, a staple of application programming for years. It provided a user interface format language and library, a data access and storage format, plus a collection of routines to provide the traditional library support, such as math and accounting. Both a language and a platform wrapped into a single environment, FoxPro was a classic example of how a programmer could learn one language/platform and remain successful and, more importantly, gainfully employed.
Then came specialization—new languages and tools, each with a particular and specific purpose. Relational databases became the norm, and SQL became the language used to access them and modify them. GUI client development initially shifted to procedural languages like C and Pascal, then, with the advent of object-orientation came C++ and Delphi. Languages such as Perl and Python became powerful tools for administering a console-based operating system, most commonly UNIX-based systems, extending the capabilities found in command shells such as the Bash and Korn shells. Then came the Web, and with it HTML, CSS, and JavaScript became the languages of choice for user interface display.
The programming language world has never seen a quiet moment, but in recent years, a large number of languages has exploded onto the programming scene, including procedural languages such as Ruby, Windows PowerShell and Python, and functional languages such as F# and Scala. Existing languages are getting new features, such as the data querying facilities Microsoft has added to C# with LINQ and now a new phase of programming language development has begun in which new custom languages, known as domain-specific languages (DSLs), are being developed for particular tasks or applications. For an example of a DSL, see the Service Station column from the 2008 Visual Studio Launch edition of MSDN Magazine (msdn.microsoft.com/magazine/cc164250).
Everywhere you turn, new languages are popping up. While multilingual programmers certainly predate it, the term "polyglot programming" originated with Neal Ford's December 2006 blog post (memeagora.blogspot.com/2006/12/polyglot-programming.html), titled (not surprisingly) "Polyglot Programming."
Polyglot Programming
OK, so we've established that there are a lot of languages out there, and there is likely one language especially suited to each problem you need to address. The savvy .NET developer needs to understand how they all fit together. That will be the focus of this column.
Consider one of the most common problems today. Developers are being asked to scale their programs—their Web sites and services, in particular—to greater and greater numbers of users than ever before. Customers are being given access to their own account information directly. This means that where an application might have had to scale to a couple of hundred users (say, to increase the number of call center employee users during a growth spurt), now the same application must scale to potentially thousands, if not millions of users.
Developers working on the Web site must manage to achieve both thread safety and good performance. Simply locking every method in order to serialize access through the entire system is not a good solution because it will not scale. Properly managing concurrency, especially as the application scales up, is no easy feat; it keeps even senior developers up late at night. Even with the release of new tools like PEX (research.microsoft.com/projects/pex) and CHESS (research.microsoft.com/projects/chess) to perform static code analysis and execute unit testing permutations to uncover multithreading bugs, you are still required to control concurrency yourself, and at a fairly low level, using either C# "lock" statements or the various concurrency-control classes from System.Threading.
Are you up to the challenge? How good are your concurrency skills? Quick, what's the difference between System.Threading.Monitor, System.Threading.Mutex, and System.Threading.Semaphore?
Here you should begin to see the value of a particular language choice. Perhaps this kind of complex code would be simpler to write and maintain if you chose a functional language like F#, whose general tendencies towards immutability and zero side effects will prevent the need for explicit concurrency control, or a domain-specific language written in Ruby, which was designed to hide the concurrency details from the developers using it.
To make this idea more concrete, imagine that a Web application needs to do some traditionally synchronous operation, such as performing some file-based I/O (or making a database or Web service call). Normally, the easiest thing to do from C# code will be to open the file via a traditional using statement, read the contents, store them to a byte array, then close the file, in a fashion similar to this:
byte[] pixels = null BinaryReader br = new BinaryReader(new FileStream(filename, FileMode.Open)); pixels = br.ReadBytes(4096);
It may be simple, but it's also horribly serialized. No additional processing can happen on this thread while the file I/O operation is taking place. (For one simple file I/O operation, this is probably not a major concern, at least not until the number of these operations gets really large, which might happen as the site tries to scale up.) It would be better to read said file using the asynchronous operations available via the CLR thread pool, as you see in Figure 1.
Figure 1 Read File Asynchronously
delegate byte[] AsyncOpenMethod(string filename); static byte[] AsyncOpen(string filename) { byte[] pixels = null; using (BinaryReader br = new BinaryReader(new FileStream(filename, FileMode.Open))) { Pixels = br.ReadBytes(4096); } } static void AsyncOpenTheFile(string filename) { byte[] pixels = null; AsyncOpenMethod aom = new AsyncOpenMethod(Class1.AsyncOpen); IAsyncResult iar = aom.BeginInvoke(filename, null, null); while (iar.IsCompleted == false) { // Do something? } pixels = aom.EndInvoke(iar); }
But the code that was simple before is now code that only a developer's mother could love, and clearly deals with much more than merely reading a file. Now examine what writing a similar routine in F# would look like:
async { use inStream = File.OpenRead(filename) let! pixels = inStream.AsyncRead(4096) }
I don't want to dive too deeply into how the F# code does this, but the let! expression tells the F# compiler to generate the expression as an asynchronous expression, and the AsyncRead method is one that F# silently tacks on to the standard System.IO.Stream-derived classes through an F# language feature called extension methods. It effectively reads the file asynchronously and dumps the results into the byte array on the left-hand side of the expression, all without any further code required from the developer.
Let's put this into a real-world context. As part of a nightly maintenance cycle, a Web service must make copies of a number of files for easier storage offline for backup purposes. The F# code to perform this file copy, all completely asynchronously, looks like Figure 2.
Figure 2 Make Backup Copies
#light open System.IO let CopyFileAsync filename = async { use inStream = File.OpenRead(filename) let! pixels = inStream.AsyncRead(4096) use outStream = File.OpenWrite(filename + ".back") do! outStream.AsyncWrite(pixels) } let tasks = [ for i in 1 .. 10 -> CopyFileAsync("data" + i.ToString()) ] let taskResults = Async.Run (Async.Parallel tasks)
Here, both the read and write will be performed asynchronously, and the entire collection of 10 tasks (one for each data file) will also be processed asynchronously. When you consider what this would take to do in C#, which would you rather write? (Chance Coble goes into vastly more detail about F# and asynchronous programming in "Easy Async: Build Concurrent Apps from Simple F# Expressions" in the October 2008 issue of MSDN Magazine.
Polyglot in Practice
In practice, interoperating between F# and C# (or any other CLR language) is relatively straightforward, once the "shape" of the code (what the language turns into at the IL level) in both languages is well understood. On the surface this seems pretty easy—after all, how different can a method call be between two languages, given that the common language specification (CLS) dictates a lot of the otherwise-awkward issues, such as parameter placement, basic types, and byte ordering?
Let's put that to the test. Taking some simple F# code to start with, let's compile it and turn it into a DLL and use ILDasm (the intermediate language disassembler, or Reflector, whichever tool you feel more comfortable with) to examine what it looks like. You can then graduate to more complicated F# expressions, such as the async workflow code Chance Coble presented in his October 2008 article, which I mentioned earlier.
To begin with, take some simple F# code
let x = 2
If we assume it lives in the default "F# Library" project file called Module1.fs, the IL in Figure 3 (with a few things commented out to keep the IL listing brief) will be generated for it.
Figure 3 The IL behind Let x=2
.assembly Library1 { .custom instance void [FSharp.Core]Microsoft.FSharp.Core. FSharpInterfaceDataVersionAttribute::.ctor(int32, int32, int32) = ( 01 00 01 00 00 00 09 00 00 00 06 00 00 00 00 00 ) // ... } .class public abstract auto ansi sealed beforefieldinit Module1 extends [mscorlib]System.Object { .custom instance void [FSharp.Core]Microsoft.FSharp.Core. CompilationMappingAttribute::.ctor(valuetype [FSharp.Core]Microsoft. FSharp.Core.SourceConstructFlags) = ( 01 00 07 00 00 00 00 00 ) .method public static int32 get_x() cil managed { // Code size 6 (0x6) .maxstack 4 IL_0000: ldsfld int32 '<StartupCode$Library1>'.$Module1::x@3 IL_0005: ret } // end of method Module1::get_x .method private specialname rtspecialname static void .cctor() cil managed { // Code size 13 (0xd) .maxstack 3 IL_0000: ldc.i4.0 IL_0001: stsfld native int '<StartupCode$Library1>'.$Module1::_init IL_0006: ldsfld native int '<StartupCode$Library1>'.$Module1::_init IL_000b: pop IL_000c: ret } // end of method Module1::.cctor .property int32 x() { .custom instance void [FSharp.Core]Microsoft.FSharp.Core. CompilationMappingAttribute::.ctor(valuetype [FSharp.Core]Microsoft. FSharp.Core.SourceConstructFlags) = ( 01 00 09 00 00 00 00 00 ) .get int32 Module1::get_x() } // end of property Module1::x } // end of class Module1 .class private abstract auto ansi sealed beforefieldinit '<StartupCode$Library1>'.$Module1 extends [mscorlib]System.Object { .field static assembly native int _init .custom instance void [mscorlib]System.Runtime.CompilerServices. CompilerGeneratedAttribute::.ctor() = ( 01 00 00 00 ) .field static assembly initonly int32 x@3 .method private specialname rtspecialname static void .cctor() cil managed { // Code size 8 (0x8) .maxstack 3 IL_0000: nop IL_0001: ldc.i4.2 IL_0002: stsfld int32 '<StartupCode$Library1>'.$Module1::x@3 IL_0007: ret } // end of method $Module1::.cctor } // end of class '<StartupCode$Library1>'.$Module1
If you trace the code through the IL, you notice two things; first, that the name "x" from the F# code is bound at the CLS level as a static property of a class called Module1, and second, that its initial value of 2 is not bound as a constant, but is initialized when the assembly is loaded, via the type constructor (.cctor) of the compiler-generated StartupCode$Library1 class. In other words, whereas you might be tempted to think of x as a constant value that the compiler can inline, the compiler chooses to present it as a static property.
This means, then, that accessing this binding, x, will require C# code something like the following:
namespace ConsoleApplication1 { class Program { static void Main(string[] args) { Console.WriteLine("F#'s x = {0}", Module1.x); } } }
So far, so good. But x is a pretty simple binding, and as such, you would expect it to be a simple matter of accessing it. Something a bit more complicated could present some trickier problems, so let's do something just a touch more complicated to make sure the mapping of F# to CLS still makes sense.
For the F# code
let add a b = a + b
the compiler generates the additional IL into the "Module1" class in the compiled F# DLL:
.method public static int32 'add'(int32 a, int32 b) cil managed { // Code size 5 (0x5) .maxstack 4 IL_0000: nop IL_0001: ldarg.0 IL_0002: ldarg.1 IL_0003: add IL_0004: ret } // end of method Module1::'add'
This is almost precisely what the C# version of the function would look like, so there's not a lot that needs to be explained here. Calling it is also trivial.
But one of F#'s strengths is that it treats functions as first-class values, and this becomes more apparent (and more difficult) when you start building functions that take functions as arguments, such as the following equivalent of the built-in F# library map function, which takes as arguments a list, along with a function to apply to each element of the list, and returns a new list containing the results:
let mymap (l : 'a list) (f : 'a -> 'b) = List.map f l
This is a particularly "functional" way of processing a list: rather than iterating through it item-by-item, a functional language takes a function as a parameter and applies it to each element in the list, generating either a single result (called a "fold" operation) or a new list containing the results of each (our "map").
Notice in Figure 4 how this turns into a rather complex bit of IL after compilation. Again, the fact that it maps to a public static method on the class Module1 is not surprising; what will make this difficult to interact with from C#, however, is that the static method takes F# lists (meaning, type-parameterized instances of Microsoft.FSharp.Collections.List) as input and return types, along with a function (meaning, an instance of a dually-type-parameterized instance of Microsoft.FSharp.Core.FastFunc) as the second input.
Figure 4 The IL for the Mapping Function
.method public static class [FSharp.Core]Microsoft.FSharp.Collections. List'1<!!B> mymap<A,B>(class [FSharp.Core]Microsoft.FSharp.Collections. List'1<!!A> l, class [FSharp.Core]Microsoft.FSharp.Core. FastFunc'2<!!A,!!B> f) cil managed { // Code size 11 (0xb) .maxstack 4 IL_0000: nop IL_0001: ldarg.1 IL_0002: ldarg.0 IL_0003: tail. IL_0005: call class [FSharp.Core]Microsoft.FSharp.Collections. List'1<!!1> [FSharp.Core]Microsoft.FSharp.Collections. ListModule::map<!!0,!!1>(class [FSharp.Core]Microsoft. FSharp.Core.FastFunc'2<!!0,!!1>, class [FSharp.Core]Microsoft.FSharp.Collections.List'1<!!0>) IL_000a: ret } // end of method Module1::mymap
This is going to require some serious C# code to work correctly. For this example, I want some C# code to take a collection of integers, and convert them into their corresponding string forms. (The fact that I could do this easily enough from within C# is irrelevant here—I have to figure out how to do the simple things before I can tackle the complex.) To call this from C# means that several things are going to have to happen successfully: the input collection will need to be converted to an F# list type; the function applied to each element must be converted into an F# "FastFunc" instance; and the returned F# list will need to be converted into a type C# can use, or else used in its native F# form directly.
The C# code is going to need those F# types, and so the first step will be to add the appropriate F# assembly reference, in this case, FSharp.Core.dll. Constructing an F# list, however, is not like constructing a C# List—rather than passing in the C# collection via a constructor, F# assumes that lists are built up using the "cons" operator, which is a static method on the F# List<> class. In other words, the F# code
let l1 = [1; 2; 3;]
turns into the rather ugly IL shown in Figure 5.
Figure 5 Let l1 = [1; 2; 3;]
IL_0000: ldc.i4.1 IL_0001: ldc.i4.2 IL_0002: ldc.i4.3 IL_0003: call class [FSharp.Core]Microsoft.FSharp.Collections. List'1<!0> class [FSharp.Core]Microsoft.FSharp.Collections. List'1<int32>::get_uniq_Empty() IL_0008: newobj instance void class [FSharp.Core]Microsoft.FSharp. Collections.List'1/_Cons<int32>::.ctor(!0, class [FSharp.Core]Microsoft.FSharp.Collections.List'1<!0>) IL_000d: newobj instance void class [FSharp.Core]Microsoft.FSharp. Collections.List'1/_Cons<int32>::.ctor(!0, class [FSharp.Core]Microsoft.FSharp.Collections.List'1<!0>) IL_0012: newobj instance void class [FSharp.Core]Microsoft.FSharp. Collections.List'1/_Cons<int32>::.ctor(!0, class [FSharp.Core]Microsoft.FSharp.Collections.List'1<!0>)
This means that to convert a C# array into an F# list suitable for passing in to the mymap method, I have to repeatedly call the static Cons method on List, passing in the new head each time; note that because the item added goes to the head of the list, to keep the F# list in the same order as the C# array, I have to start at the end of the array and work backward:
int[] scores = {1, 2, 3, 4, 5}; var fs_scores = Microsoft.FSharp.Collections.List<int>.get_uniq_Empty(); for (int i = scores.Length-1; i >= 0; i--) { fs_scores = Microsoft.FSharp.Collections.List<int>.Cons(scores[i], fs_scores); }
Already this is becoming something of a pain. Obtaining an instance of F#'s FastFunc type is even more tedious because the FastFunc type, like the core System.Delegate type, isn't really intended to be instantiated by programmers, but instead handled by the F# compiler. Ordinarily, when constructing an instance of this in F# code, the compiler actually generates an inner class that inherits from FastFunc, something you could do from C#, but that would require a lot more work.
Suddenly calling into this F# function seems like way too much work for the benefit derived.
What all of this serves to illustrate is that the mapping of a language to its underlying platform is crucial to deciding where and how to use languages in a polyglot manner. Something that is trivial to do inside of F#, for example, can turn out to be difficult to do directly from C#. This doesn't mean giving up on the idea of polyglotism, it just means you must carefully choose which languages you use together and for what purposes. From a more positive perspective, consider, one of the examples from Chance's October article with additional code at the bottom (see Figure 6)
Figure 6 The Original Code Plus Modifications
open System open System.IO open Microsoft.FSharp.Control.CommonExtensions #nowarn "057" let aCountSpace filename size = async { let space = Convert.ToByte ' ' use stream = File.OpenRead (filename) let bytes = Array.create size space let! nbytes = stream.ReadAsync (bytes,0,size) let count = bytes |> Array.fold_left (fun acc x -> if (x=space) then acc + 1 else acc) 0 return count } let aCounted (files : FileInfo array) = files |> Array.map (fun f -> aCountSpace (f.FullName) (int f.Length)) |> Async.Parallel |> Async.Run
Note that I've modified it slightly. However, the only modification I made here was to specify the "files" argument in the "aCounted" method to be an array of FileInfo, something which was inferred by the F# compiler in Chance's original code. The IL signature for "aCounted" now looks like
.method public static int32[] aCounted(class [mscorlib]System.IO.FileInfo[] files) cil managed
which makes it pretty trivial to call from C#, as in:
int[] fileCounts = Module1.aCounted(new DirectoryInfo(@"C:\Projects\Test").GetFiles("*.txt")); foreach (var ct in fileCounts) { Console.WriteLine("Count = {0}", ct); }
Yet this new version, as simple as it is to call, retains the full asynchronous execution capability that Chance discussed back in October.
Note that the use of C# here is purely arbitrary, based entirely on the fact that I feel more comfortable in C# than in Visual Basic or C++/CLI; having said that, it shouldn't be difficult to see how the same code would look in either of those languages.
The Cost behind the Choices
There's always a cost to taking a polyglot approach, of course. The first major headache is pretty obvious to see, since we've already run into it—developers who want to make use of different languages on a project must understand how those languages map to the underlying platform—and subsequently to each other. This means that simply relying on the compiler to make things right will no longer work.
Developers will need to see how the compiler maps the language constructs to the underlying platform (in this case, the CLR), and how other languages will pick up those constructs. The good news here derives from the fact that languages don't change much over time, even for languages that have seen some pretty radical shifts over the last decade, such as C# or Visual Basic, so once you've learned some of the language mappings, this information remains relatively constant.
The second problem derives from the first—debugging a polyglot program can be more difficult than debugging a monoglot one, owing simply to the fact that now the developer must speak two (or more) languages, rather than just the one. In the case of the code I've presented, for example, debugging will require setting a breakpoint in the C# code and stepping through, potentially into the F# code and back again. In many respects, this is nothing new—F# is just one more language on to the list of languages you need to know.
In some development shops, the developers working on the C# code simply trust (either by declaration or through faith in unit tests) that the F# code works as advertised, and step over rather than step into the F# code, just as many Visual Basic and C# developers do when stepping through code that calls into unmanaged C or C++ libraries.
In future columns, I'll look at other ways that polyglot approaches make programming easier, and at other languages and their benefits to the polyglot programmer.
Send your questions and comments for Ted to polyglot@microsoft.com.
Ted Neward is a Principal Consultant with ThoughtWorks, an international consultancy specializing in reliable, agile enterprise systems. He has written numerous books, is a Microsoft MVP Architect, INETA speaker, and PluralSight instructor. Reach Ted at ted@tedneward.com, or read his blog at blogs.tedneward.com.