House of COM: Migrating Native Code to the .NET CLR

Article
02/26/2008

Migrating Native Code to the .NET CLR

Don Box

espite the hopes of management, shareholders, or overly-ambitious developers, it is simply not possible to get every piece of software ever written to run as managed code in the Microsoft® .NET common language runtime (CLR). Given the shortage of programming talent, it is unlikely that many organizations will rewrite 100 percent of their existing Win32®-based, C, C++, Visual Basic® 6.0-based, or COM codebase no matter how compelling the new runtime may be. That means that the world of native code written before the .NET era needs to peacefully coexist with new managed components written in .NET-friendly languages. Thankfully, the CLR supports bridging the two worlds without undue pain or suffering.
To allow native and managed code to coexist, some mechanism is needed to allow managed code executing inside of the CLR (MSCOREE.DLL) to integrate with code running outside of MSCOREE.DLL. This means that managed programs need to run in "unmanaged mode" long enough to execute classic COM or Win32-based code. Also, unmanaged threads may need to visit MSCOREE.DLL long enough to be able to invoke a method on a managed type.

Figure 1 Integrating Managed and Unmanaged Code

Fortunately, the CLR supports both types of transitions, allowing both managed and native DLLs to coexist in a single process and call from one DLL to another independent of whether or not they are hosted inside the CLR. An example of a typical circa-2001 process is shown in Figure 1. Note that some code runs inside of MSCOREE.DLL and some does not. Unless the entire operating system is rewritten in managed code, this is likely to be the state of affairs for some time, as today, the underlying Windows® operating system is exposed via native DLLs, not managed DLLs.

P/Invoke

      Calling a native Win32-based DLL from the CLR is fairly straightforward using P/Invoke (the P stands for platform). P/Invoke is a technology that allows you to map a static method declaration to a PE/COFF entry point that is resolvable via LoadLibrary/GetProcAddress. Like the Java Native Interface (JNI) and J/Direct® before it, P/Invoke uses a managed method declaration to describe the stack frame, but assumes the method body will be provided by an external, native DLL. Unlike JNI, however, P/Invoke is useful for importing "heritage" DLLs that were not written with the CLR in mind.
      To indicate that a method is defined in an external, native DLL, you simply mark the static method as extern and use the System.Runtime.InteropServices.DllImport method attribute. The DllImport attribute tells the CLR which arguments to pass to LoadLibrary and GetProcAddress when it is time to call the method. The built-in C# DllImport attribute is simply an alias for System.Runtime.InteropServices.DllImport.
      The DllImport attribute takes a variety of parameters. As shown in Figure 2, the DllImport attribute requires at least a file name to be provided. This file name is used by the runtime to call LoadLibrary prior to dispatching the method call. The string to use for GetProcAddress will be the symbolic name of the method unless the EntryPoint parameter is passed to DllImport. Figure 3 shows two ways to call the Sleep method in kernel32.dll. The first example relies on the name of the C# function matching the name of the symbol in the DLL. The second example relies on the EntryPoint parameter instead.
      When calling methods that take strings, you need to set the Unicode/ANSI policy either on the method or on the surrounding type. This is needed to control how string types are translated for consumption by unmanaged code. The CharSet parameter to DllImport allows you to specify whether Unicode (CharSet.Unicode) or ANSI (CharSet.Ansi) should always be used, or if the underlying platform should automatically decide based on Windows NT® or Windows 2000 versus Windows 9x or Windows Millennium Edition (Me) (CharSet.Auto). Using CharSet.Auto is similar to writing Win32-based code in C using the TCHAR datatype, except that the character type and API is determined when loading, not at compile time, allowing a single binary to work properly and efficiently on all versions of Windows.
      The Windows platform has a variety of name mangling schemes to indicate calling convention and character sets. When the CharSet is set to CharSet.Auto, the symbolic name will automatically have a W or A suffix, depending on whether Unicode or ANSI is used by the runtime. Additionally, if the plain symbol is not found, the runtime will munge the symbol using the stdcall conventions (for instance, Sleep may be _Sleep@4). This symbolic mangling can be suppressed using the ExactSpelling parameter to the DllImport attribute.
      Finally, when calling Win32 functions that use COM-style HRESULTs, you have two options. By default, P/Invoke treats the HRESULT as simply a 32-bit integer that is returned from the function, requiring the programmer to manually test for failure. A more convenient way to call such a function is to pass the TransformSig=true parameter to the DllImport attribute. This tells the P/Invoke layer to treat that 32-bit integer as a COM HRESULT and to throw a COMException in the face of a failed result. (Note that the TransformSig parameter from Beta 1 will be renamed PreserveSig in Beta 2, and its meaning is reversed from Beta 1.) Given the two declarations shown in Figure 4, calling OLE32Wrapper.CoImpersonateClient1 requires the programmer to manually check the result and deal with failed HRESULTs explicitly. Because the TransformSig parameter was used, calling OLE32Wrapper.CoImpersonateClient2 tells the CLR to map failed HRESULTs to COMExceptions implicitly without programmer intervention. In the case of OLE32Wrapper.CoImpersonateClient2, the method returns void because there is no returned value from the underlying function to deal with. Had the method been declared to return a typed value (such as double), the P/Invoke layer would have assumed that the underlying native function accepted an additional parameter by reference (a la COM's [retval] attribute). This mapping only takes place when the TransformSig parameter is true.

Typed Transitions

When calling out of (or into) the runtime, parameters are invariably passed on the call stack, as shown in Figure 5. These parameters are instances of types both to the runtime and to the outside world. The key to understanding how interop works is to understand that any given "value" has two types: a managed type and an unmanaged type. More importantly, some managed types are isomorphic to an unmanaged type, which means that when an instance of that type needs to be passed outside of the runtime, no conversion is necessary. However, many types are not isomorphic and require some conversion in order to come up with a representation that is suitable for the outside world.

Figure 5 Parameters on the Call Stack

Figure 6 shows a list of the basic isomorphic and nonisomorphic types. When calling an external routine that takes only isomorphic types as parameters, no conversion is needed and the caller and callee can actually share a stack frame despite the fact that one side is running in unmanaged mode. If at least one parameter is a nonisomorphic type, the stack frame must be marshaled into a format compatible with the world outside of MSCOREE. That conversion may need to take place in the other direction for parameter values passed from the function back to the caller. Fortunately, the C# compiler knows which direction parameter values flow based on the ref and out keywords. Figure 7 shows this effect.

Figure 7 Nonisomorphic Parameters

You are free to control how a given parameter (or field in a struct) marshals using the MarshalAs attribute. This attribute indicates which unmanaged type should be presented to the world outside MSCOREE. At the very least, you can use the MarshalAs attribute to indicate what the corresponding native type is for a given parameter or field. For many types, the CLR will pick a reasonable default. However, you can override these defaults using the MarshalAs attribute. For example, the code in Figure 8 shows how the MarshalAs attribute can be used to map the CLR System.String type to one of the four common Win32 formats. As a point of interest, all but the UnmanagedType.LPWStr parameter will result in a copy of the string being created to comply with the underlying native format.
In addition to using the MarshalAs attribute to control type mappings on a field-by-field or parameter-by-parameter basis, you can also control the underlying representation of structs and classes. In particular, the StructLayout and FieldOffset attributes allow you to precisely control the in-memory layout of classes and structs, which is critical for structs that are passed outside of the runtime. The following is an example of an annotated C# struct.

  using System.Runtime.InteropServices;
[ StructLayout(LayoutKind.Sequential) ]
public struct PERSON_REP {
  [ MarshalAs(UnmanagedType.BStr) ] 
  public String name;
  public double age;
  [ MarshalAs(UnmanagedType.VariantBool) ]
  bool dead;
}

This code is the equivalent definition in COM IDL.

  struct PERSON_REP {
  BSTR name;
    public double age;
    VARIANT_BOOL dead;
  };

RCW and CCW

When passing object references other than System.String or System.Object, the default marshaling behavior is to convert between CLR object references and COM object references. As shown in Figure 9, when a reference to a CLR object is marshaled across the MSCOREE boundary, a COM-callable wrapper (CCW) is created to act as a proxy to the CLR object. Likewise, when a reference to a COM object is marshaled in through the MSCOREE boundary, a runtime-callable wrapper (RCW) is created to act as a proxy to the COM object. In both cases, the "proxy" will implement all of the interfaces of the underlying object. Additionally, the "proxy" will try to map COM and CLR idioms like IDispatch, object persistence, and events to the corresponding construct in the other technology.

Figure 9 RCW and CCW Architecture

For interfaces that straddle the MSCOREE boundary via RCWs or CCWs, the CLR relies on a set of annotations to the managed interface definition to give the underlying marshaling layer hints as to how to translate the types. These hints are a superset of those just described for P/Invoke. Additional aspects that need to be defined include UUIDs, vtable versus dispatch versus dual mappings, how IDispatch should be handled, and how arrays are translated. These aspects are added to the managed interface definition using attributes from the System.Runtime.InteropServices namespace. In the absence of these attributes, the CLR makes conservative guesses about what the default settings for a given interface and method should be. For new managed interfaces that are defined from scratch, it is useful to use the attributes explicitly if you intend your interfaces to be used outside of the common language runtime.

TLBIMP and TLBEXP

Translating native COM type definitions (such as structs, interfaces, and so on) to the CLR can be done by hand, and in some cases this is necessary, especially when no accurate TLB is available. Translating type definitions in the other direction is simpler given the ubiquity of reflection in the CLR, but again, you are better off using a tool than resorting to hand translations. The CLR ships with code that does a reasonable job of doing this translation for you, provided that COM TLBs are accurate enough. System.Runtime.InteropServices.TypeLibConverter can translate between TLBs and CLR assemblies. The ConvertAssemblyToTypeLib method reads a CLR assembly and emits a TLB containing the corresponding COM type definitions. Any hints to this translation process (such as MarshalAs) must appear as custom attributes on the interfaces, methods, fields, and parameters in the source types. The ConvertTypeLibToAssembly method reads a COM TLB and emits a CLR assembly containing the corresponding CLR type definitions. The SDK ships with two tools (TLBIMP.EXE and TLBEXP.EXE) that wrap these two calls behind a command-line interface suitable for use with NMAKE. The relationship between these tools is shown in Figure 10.

Figure 10 TLBIMP and TLBEXP

In general, it is easier to first define types in a CLR-based language and then emit the TLB. For example, consider the C# code shown in Figure 11. If you were to treat this code as a pseudo-IDL file, you could run it through csc.exe and tlbexp.exe to produce a TLB that is functionally identical to the one produced by the real IDL file shown in Figure 12. The advantage to using the C# approach is that the type definitions are extensible and easily readable, neither of which could be said for the TLB or IDL file.

REGASM

Some developers want their CLR classes to be accessible via COM's CoCreateInstance. For this to happen, some registry entries need to be inserted that bind a COM CLSID to your assembly and type. The REGASM.EXE tool does just this. REGASM.EXE takes an assembly as an argument and writes the appropriate registry entries for each public class. The InprocServer32 entry points to MSCOREE.DLL, which acts as the COM facade to your CLR-based objects. Be aware that REGASM.EXE is not strictly necessary, as you can fairly easily host the CLR from native code and use the default AppDomain to load types and instantiate new instances of any class. Figure 13 shows an example of this using the System.Collections.Stack class from Visual Basic 6.0. This can be further automated using a moniker. Surfing to http://discuss.develop.com/archives/wa.exe?A2=ind0008&L=DOTNET&P=R63059 describes the ".NET moniker" that allows you to instantiate any CLR type using classic COM's CoGetObject or the GetObject method in Visual Basic 6.0.

Strategies

There are three basic strategies for moving native code into the runtime. Figure 14 illustrates the partial port approach, in which source code is mechanically translated from a pre-CLR language (such as Visual Basic 6.0) to a CLR-aware language (such as Visual Basic.NET or C#). This approach relies on the fact that any subordinate native libraries can be imported into the CLR using interop services.

Figure 14 .NET Via Partial Source Code Porting

Figure 15 illustrates the full port approach, in which source code is completely rewritten in a CLR-aware language without relying on any native subcomponents. This approach relies on the existence of managed versions of any subordinate native libraries. It also relies on your willingness to adapt source code to any stylistic differences the managed libraries may mandate. This approach requires the most labor, but has the advantage of fewer interop transitions as well as (typically) richer and better-designed libraries.

Figure 15 .NET Via Full Source Code Porting

Figure 16 illustrates the "punt" approach, in which source code is left alone and the code is imported into the CLR using interop services. This approach does not rely on the existence of managed versions of any subordinate native libraries, nor does the developer need to adapt source code in any significant manner. This approach requires the least labor, and may have the advantage of fewer interop transitions due to the ratio of subordinate library calls per method.

Figure 16 .NET Via Binary Import (Punt Approach)

While you could always use technologies like XML or SOAP, the CLR provides developers a variety of choices for transitioning from the old world of COM and Win32 to the new world of C# and Web Services.

Send questions and comments for Don to housecom@microsoft.com.

Don Box is a cofounder of DevelopMentor, providing education and support to the software industry. Don wrote Essential COM, and cowrote Effective COM and Essential XML (all Addison-Wesley). He coauthored the W3C SOAP spec. Reach Don at http://www.develop.com/dbox. This column is adapted from the forthcoming book on .NET programming with C#, co-authored by Don Box and Ted Pattison (Chapter 1), © 2002. Use is by permission of Pearson Education Inc. All rights reserved.

From the May 2001 issue of MSDN Magazine