|
s you may have guessed from looking at the top of this page, the House of COM column that appeared in the May 2001 issue was my last column on COM (I have already started the bidding war on eBay for that soon-to-be-collector's edition). Dedicated readers may have noticed that I haven't been using the column to write about COM per se for some time. Rather, for the past three years, my work has been shifting to encompass other software integration/component technologies, such as the Common Language Runtime (CLR), ISAPI, XML, SOAP, and Web Services.
Acknowledging the fact that everyone who ever needed to understand the subtle ramifications of CoCreateFreeThreadedMarshaler on the global economy has already either retired from the industry or has moved onto management, I am officially transitioning from COM to more modern technologies, broadly covered by the umbrella term "Web Services." Because the phrase Web Service is being used to sell everything from underarm deodorant to tax-cut plans, I'll start by fleshing out the term. For the purposes of this column, the term Web Service will mean the following: software that is exposed to other software over Internet-friendly protocols. Short, sweet, and to the point. I promise to use the term only when it applies to the preceding definition. Honest.
Web Services typically use HTTP as their transport, although other Internet-friendly protocols (such as SMTP) can be used. Web Services typically use XML as their data format, although other MIME-friendly formats are possible. Web Services differ from traditional Web pages or applications primarily based on the target audience. Traditional Web pages and applications target carbon-based life forms (humans). Web Services target silicon-based life forms (software agents).
This first column dives right into the fray by looking at the support for XML Schemas in the Microsoft® .NET platform. XML Schemas are a critical component for Web Service infrastructure, as SOAP relies on the XML Schema type system for its data model and WSDL (Web Services Description Language) relies on the XML Schema language for defining message formats. The focus of this month's column is on the XML serializer and schema compiler. So, without further ado....
What is XML Schema?
XML provides structure and type over data. Classically, XML's strength has been providing structure, both in terms of its transfer syntax (that is, the angle brackets we all know and love) and in terms of its abstract data model, the Infoset, upon which all other XML technologies are based. However, prior to the advent of XML Schemas, XML lacked a type system capable of addressing the needs of software integration technologies such as SOAP or Web Services. Now that the XML Schema specification has been ratified as a W3C recommendation, virtually every major XML vendor (including Microsoft) is burning the midnight oil to integrate XML Schema support into their platforms, applications, and tools.
XML Schemas enable smart editors, IntelliSense®, code generation, and general-purpose validators. The XML Schema specification is partitioned into two specifications: Part 1 specifies a language for defining composite types (called complex types) that describe the content model and attribute inventory of an XML element. In addition to this type definition language, Part 1 of the XML Schema specification describes how XML Schemas influence the abstract data model of an XML document. In particular, Part 1 describes a family of reflection-like augmentations that schema-aware software must expose to consuming applications. These augmentations are collectively known as the post-schema validation infoset (PSVI). Part 2 specifies a set of built-in primitive types and a language for defining new primitive types (called simple types) in terms of existing types. In addition to Parts 1 and 2, there is a primer to the XML Schema language known as Part 0 that provides an excellent overview of XML Schemas.
Dedicating the time and energy needed to understand Parts 0-2 of the XML Schema specification is a valuable exercise for anyone working in XML, and particularly those working in SOAP and Web Services. If you don't have a free month to spare this year, the following overview of the XML Schema language will have to suffice. Readers who are already familiar with XML Schemas should feel free to skip to the next section.
The XML Schema language is a rich XML-based vocabulary for describing the type affiliation of XML elements and attributes. The following is a simple schema document that contains exactly one global element declaration:
<xsd:schema
targetNamespace="urn:xyzzy"
xmlns:target="urn:xyzzy"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
>
<xsd:element name="bob" type="xsd:double" />
</xsd:schema>
This schema document indicates that the element named bob, whose namespace URI is urn:xyzzy, has a child of type double as defined in Part 2 of the XML Schema specification. The following is an example of the corresponding instance document:
<ns:bob xmlns:ns="urn:xyzzy">33</ns:bob>
Schema-aware processing software will treat the previous instance document as identical to one of the following instance documents:
<ns:bob xmlns:ns="urn:xyzzy">33.0000</ns:bob>
<ns:bob xmlns:ns="urn:xyzzy">3.3e1</ns:bob>
Even though these three instance documents are lexically different, schema-aware software will treat them as identical. This is one of the side-effects of working in the PSVI-based world of schemas.
Most schema documents define new types rather than simply applying the existing types. The schema type system supports two forms of type definition: simple types and complex types. Simple types can only be represented as text with no markup. Complex types can be represented using element and/or character data children and attributes. To that end, a complex type can only be applied to an element. In contrast, simple types may be applied to both elements and attributes.
The XML Schema type system is hierarchical and supports inheritance and substitution. Figure 1 shows the type system of XML Schema, including the set of built-in types defined in Part 2 of the schema specification. Note that the root of the type system is anyType, which when applied to an XML element or attribute, allows any content and, in the case of an element, allows any attribute to be applied. Element declarations with no explicit type specified are implicitly of type anyType. Attribute declarations with no explicit type specified are implicitly of type anySimpleType.
.gif)
Figure 1 XML Schema Simple Types
Figure 2 shows a non-trivial schema document that defines two complex types. The first type, Person, describes an element with exactly two child elements. The first child element of Person (described by the xsd:choice particle) is of type string and is named either name or id. The second child element of Person (described by the xsd:any particle) may have any name and be affiliated with any type. These two elements are aggregated under a sequence compositor (described by the xsd:sequence element) that indicates that the name or id element must appear first, followed by the wildcard element.
The second type in the schema, AgedPerson, derives from Person by extension. This means that the content model specified in the AgedPerson definition is interpreted as following the content model of the base type Person. So an element whose type is AgedPerson will have three elements. The first two elements will be exactly as described for Person. The third element will be of type double and will be named either age or timeOnEarth. Finally, you should note that the schema document ends with a global element declaration that binds the element named don to the type named Person.
The following snippet shows an instance document that corresponds to the schema document just described.
<ns:don
xmlns:ns="uuid:048b2fa1-d557-473f-ba4c-
acee78fe3f7d"
>
<name>Don Box</name>
<niceStuffForDon/>
</ns:don>
The type of the root element is Person due to the global element declaration for the element named don.
Because the XML Schema type system supports substitution based on inheritance, you can easily indicate that the don element actually contains an AgedPerson rather than the expected Person. This is done using the xsi:type attribute shown here.
<ns:don
xmlns:ns="uuid:048b2fa1-d557-473f-ba4c-acee78fe3f7d"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:type="ns:AgedPerson"
>
<name>Don Box</name>
<niceStuffForDon/>
<age>26</age>
</ns:don>
In this example, the derived type is substituted for the base type, which allows the age element to follow the initial two elements of Person. Had the xsi:type attribute not been specified, the age element would have caused a validation error upon parsing.
Finally, it is useful to look at the Person type definition one last time. Note that the name element declaration indicates that the element is "nillable." The XML Schema specification allows an instance document to indicate that it has no data using the xsi:nil attribute. The xsi:nil attribute is roughly equivalent to a SQL null, a null pointer, or reference in the programming language of your choice. For example, the following is a legal Person
<ns:don
xmlns:ns="uuid:048b2fa1-d557-473f-ba4c-acee78fe3f7d"
>
<name />
<niceStuffForDon/>
</ns:don>
as is the following:
<ns:don
xmlns:ns="uuid:048b2fa1-d557-473f-ba4c-acee78fe3f7d"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>
<name xsi:nil="true" />
<niceStuffForDon/>
</ns:don>
The name element in the first example contains a string of length zero. The name element in the second example contains nil, which is distinct from all legal string values.
Schema Support in System.Xml
As of Beta 2 of the .NET platform, programmers working in C#, Visual Basic® .NET, or any CLR-friendly language can take advantage of XML Schemas in a variety of ways. The managed XML classes (System.Xml), SOAP and Web Services, Visual Studio .NET, and the managed data access classes (System.Data) all provide varying levels of support for XML Schemas. All of these technologies layer their schema support over the System.Xml.Schema namespaces.
The System.Xml.Schema library provides a type-safe, in-memory object model for schema documents for programmers using the .NET platform. This object model closely matches the constructs of the XML Schema language. Given this type hierarchy, you could construct an in-memory schema that corresponds to the schema document shown in Figure 2. Such an in-memory schema would look like Figure 3.
.gif)
Figure 3 System.Xml.Schema Object Model
An in-memory schema can be created either by explicit manipulation of the schema object model or by loading a schema document from a disk or the network. Figure 4 shows the C# code needed to construct the schema shown in Figure 2. Assuming that the schema already exists (which is much more common), you can simply write the following four-liner:
XmlReader reader
= new XmlTextReader("http://foo.com/don.xsd");
XmlSchema schema = XmlSchema.Load(reader, null);
schema.Compile(null);
reader.Close();
Note that in both examples, a call to XmlSchema.Compile is made. This informs the underlying implementation to pre-compile an optimized in-memory representation that is used to speed up schema processing and validation. This optimization is used primarily by System.Xml.XmlValidatingReader, which is an XmlReader-based class that provides schema processing over any XML data source.
Mapping Between the XML and CLR Type Systems
XML Schema types have attribute inventory and content models, both of which augment the "value" of the type with named composite values. These named composite values are remarkably similar to the fields of a programmatic type, which causes programmers around the world to want to map back and forth between the two type systems, as shown in Figure 5. If a mapping exists between a schema type and a programmatic type, then XML serialization can be derived automatically by reflecting against both type definitions.
.gif)
Figure 5 Bridging the XML Type System
Programs that translate between programmatic and schema types are called "schema compilers." Schema compilers automatically generate serialization code based on XML Schema types, much as IDL compilers do today for DCOM and CORBA-based systems. The .NET platform ships with a schema compiler (XSD.EXE) which I'll discuss later in this piece. Some proponents of schema compilers hope that traditional hand-coded XML processing using the DOM or SAX will be rendered obsolete. While in principle, using machine-generated code is almost always a good idea, the most common obstacle to using schema compilers exclusively is the difference between the XML Schema type system and most programmatic type systems.
When mapping between the XML Schema type system and a programmatic type system, issues invariably arise due to the mismatch between in-memory and serialized data. Also, the XML Schema type system needs to address the needs of SGML-style content management systems, which means that many schema constructs make little sense in a programmatic type system. These schisms led to two fundamental approaches to bridging the two worlds: the XML Schema-centric approach and the programmatic type-centric approach (see Figure 6). The former assumes that the XML Schema type system is the dominant type system and that the concerns of the programmatic type system are subordinate. The latter approach assumes the converse, giving second-class status to the XML Schema type system.
.gif)
Figure 6 Class and Schema-centric Type Mapping
Neither the schema-centric nor the programmatic type-centric approach is ideal. If you adopt the programmatic type-centric approach there are a world of XML documents in nature that will become inaccessible without resorting to hand-coding a serializer. Typical constructs in XML that do not map cleanly to programmatic types include xsd:choice, derivation from a primitive type such as int, nillable elements whose type is a value type (not a reference type), derivation by restriction, and mixed content. Running a schema compiler on schemas that use these constructs typically results in a loss of fidelity at best and incorrect/incomplete mappings at worst.
The situation is not markedly better if you adopt the schema-centric approach to serialization. The XML Schema type system does not have intrinsic support for typed references or arrays, which means that a schema-centric approach usually results in loose approximations of the corresponding programmatic types. The SOAP specification tried to address these limitations in its encoding scheme, but unfortunately, very few SOAP implementations deal with typed arrays or references correctly, rendering them largely useless for broad adoption. This means that you can't simply run a schema compiler over your components and get an XML Schema for free. As shown in Figure 6, invariably there will be types that don't make the transition, as was the case in the schema-centric approach.
Fortunately, the .NET Framework supports both approaches by providing two XML-based serialization engines. The System.Runtime.Serialization serializer takes the programmatic type-centric view, and can take literally any type in the CLR and map it to the XML Schema type system with full fidelity. The System.Xml.Serialization serializer takes the schema-centric view, and can take a broader range of XML Schema types and map them to the CLR type system with reasonable fidelity. At the time of this writing (Beta 2 of .NET), both serializers have issues handling various XML Schema constructs, however, the System.Xml. Serialization serializer could handle a much broader range of schema. Even when .NET is ultimately shipped, the intrinsic differences between the CLR and XML Schema type systems guarantee that programmers will have a busy future dealing with these mismatches by hand.
XmlSerializer Architecture
XmlSerializer is the focal point of the System.Xml.Serialization library, and is the one type that all users of the library need to be aware of. The XmlSerializer type provides a simple and convenient interface to the library's underlying serialization engine. As shown in Figure 7, this serialization engine supports converting between CLR and XML-based instances. The serialization engine has a set of default mapping rules to determine how each field or property of a CLR-based class maps onto an XML element or attribute. These defaults can be overridden using custom attributes, which I'll discuss in detail later.
Figure 8 shows the signature of the XmlSerializer type. Note that the constructor requires the caller to provide a System.Type object. The XML serialization engine requires a distinct XmlSerializer object for each type that is in use. When either serializing or deserializing an object, it is imperative that the XmlSerializer object used to perform the serialization and deserialization be affiliated with the correct CLR type.
The XmlSerializer type has two primary methods: Serialize and Deserialize. The Serialize method writes the XML-based instance that corresponds to the provided CLR-based object. The Deserialize method reads an XML-based instance and returns the corresponding CLR-based object. Both methods require an XmlReader or XmlWriter to read or write the XML. The XmlSerializer class provides overloaded versions of these methods that accept either a TextWriter/TextReader or a Stream object. However, in both of these overloads, the underlying implementation simply creates an XmlTextReader/XmlTextWriter and delegates to the primary Serialize or Deserialize method.
Figure 9 and Figure 10 show the canonical usage of the Serialize and Deserialize methods, respectively. Please note that in both cases, the correct System.Type object must be used in order to initialize the serializer properly.
XmlSerializer manages the generation and execution of per-type reader/writer pairs. These pairs are dynamically generated types that extend the internal types XmlSerializationReader and XmlSerializationWriter, respectively. When instantiating an XmlSerializer, you must provide a System.Type object that represents the type the new serializer object will be used with. To avoid generating redundant assemblies, the XmlSerializer constructor looks in an AppDomain-wide cache of generated reader/writer assemblies and reuses the cached assembly if one is found. If one is not found, the XmlSerializer constructor generates a new dynamic assembly by reflecting against the presented System.Type object. This new assembly is added to the cache so subsequent XmlSerializer objects can reuse it, reducing overall code size and codegen overhead. Unfortunately, the cache only works for some constructors of XMLSerializer, since the lookup index gets more complex with more overrides. Users can cache the serializer themselves—it's freethreaded so you can serialize across threads.
.gif)
Figure 11 XMLSerializer Architecture
Every XmlSerializer object holds a reference to a dynamically generated assembly (see Figure 11). In most cases, this assembly is simply fetched from a cache and may, in fact, be shared by other XmlSerializer objects that are supporting the same CLR type.
Type Mapping
The XmlSerializer architecture assumes that there is an XML Schema complex type that corresponds to each CLR struct and class. Assuming no special CLR attributes are used, this complex type is assumed to have one element child per public property or field. The name of the child element matches the field/property name. The type of the child element matches the field's/property's corresponding Schema type. All of the element children are serialized as if an <xsd:sequence /> compositor were in use, which results in the child elements corresponding to the type's fields/properties appearing in the order in which they are declared. This is a reasonable default, since in all likelihood, most XML is generated by software and the flexibility offered by the <xsd:all> compositor (which was used in Beta 1) does not offset the performance impact you'll feel if you need to handle elements in any order.
The [XmlType] attribute allows you to control the name and namespace URI of the schema type that is associated with a CLR type. The [XmlType] accepts a Namespace parameter that corresponds to the XML Schema's targetNamespace attribute (see Figure 12). If this parameter is not present, then the corresponding schema type is not affiliated with any XML namespace. The TypeName parameter allows you to control the local name of the corresponding XML Schema type and, if not provided, the local name of the CLR type is used. To prevent any schema mapping from taking place, you can specify the IncludeInSchema-false parameter, not shown in Figure 12, which suppresses any XML Schema conversion of the type.
.gif)
Figure 12 XMLRootAttribute/XMLTypeAttribute Mapping
If you're going to use an XML Schema complex type as a top-level element in an XML instance document, then a global element declaration is also needed. The [XmlRoot] attribute causes such an element declaration to be created. As shown in Figure 12, the [XmlRoot] attribute allows you to control the element name and XML namespace URI. It also allows you to control the nillable attribute for supporting xsi:nil in instance documents. Finally, if the element declaration is going to use a built-in simple data type rather than a complex type, the DataType attribute can be used. Figure 13 shows each of the parameters to the [XmlType] and [XmlRoot] attributes.
Using the [XmlType] and [XmlRoot] attributes is relatively simple. Consider the C# code shown here.
using System.Xml.Serialization;
[ XmlRoot(ElementName="Bobby", Namespace="abcdef") ]
[ XmlType(TypeName="Robert", Namespace="abcdef") ]
public class Bob {}
[ XmlRoot(ElementName="Stevie", Namespace="abcdef") ]
public class Steve {}
[ XmlType(TypeName="Donald", Namespace="abcdef") ]
public class Don {}
public class Dave {}
The schema documents that this code corresponds to is shown in Figure 14. Note that because the CLR type Don does not have an [XmlRoot] attribute, the corresponding global element declaration in the schema document is based on the CLR type name and has a namespace URI of "". Also note that because the CLR type Steve does not have an [XmlType] attribute, the corresponding complex type definition simply uses the CLR type name verbatim.
.gif)
Figure 15 XSD.EXE
The System.Xml.Serialization library can automatically generate XML Schemas from a .NET assembly. It can also generate annotated source code from an XML Schema. This capability is typically accessed using the XSD.EXE tool. The XSD.EXE tool accepts either an assembly or an XML Schema and uses the rules just described to generate the type definitions in the desired type system (see Figure 15).
Schema Customization
When mapping a CLR type to an XML Schema type, each CLR field/property maps to some construct in the corresponding XML Schema type. By default, all public fields and properties are assumed to map to child elements inside an <xsd:sequence /> compositor. This default mapping can be overridden using the [XmlElement] and [XmlAttribute] custom attributes. These attributes cause the attributed field/property to map to either a local element declaration or a local attribute declaration, respectively. Both of these attributes accept parameters that customize the generated XML Schema types. These parameters are shown in Figure 16.
.gif)
Figure 17 XMLElementAttribute Mapping
The [XmlElement] attribute maps to a local element declaration in the XML Schema language. As shown in Figure 17, there are two usage models. The first model assumes a pure local element declaration, in which the namespace affiliation is controlled by the parent complex type. The second model assumes that a reference to a global element declaration will be used, and is typically only used to support element substitution groups in the XML Schema language. The [XmlAttribute] attribute works much the same way, except for XML attributes (see Figure 18).
.gif)
Figure 18 XMLAttribute Mapping
Figure 19 shows an annotated CLR type that uses both the [XmlElement] and [XmlAttribute] attributes. The corresponding schema for this type is shown in Figure 20. Note that in this example, the name element is marked form="qualified", but the notbreathing element is not. This is due to the use of the Form parameter in the [XmlElement] attribute used in the C# code.
CLR enums are treated specially when mapping to XML Schemas. A CLR enum maps to an XML Schema simple type that restricts the built-in type string to a set of enumerated values, one per CLR enum member. The values of these restrictions correspond to the CLR enum member name unless the [XmlEnum] attribute is used. Here's an annotated CLR enum:
using System.Xml.Serialization
[ XmlType(TypeName="hair", Namespace="abcdef") ]
public enum AuthorHair {
[ XmlEnum("bald") ] Harrisonish,
[ XmlEnum("short")] Gudginesque,
[ XmlEnum("medium")] Skonnarded,
[ XmlEnum("long")] Boxy,
}
Take a look at the code in Figure 21 to see the corresponding XML Schema type.
At the time of this writing, one of the weaker aspects of the mapping between the CLR and XML type systems is related to substitution and inheritance. Both type systems have their own model for derivation and substitution; however, the current implementation of the serialization engine cannot always deal with these models in a reasonable way.
The XML Schema type system supports two forms of substitution. One form uses the xsi:type attribute to indicate that a derived type is in use. This form is supported by the XML serialization engine. The other form, based on element substitution groups, is only partially supported, as the corresponding CLR type's field must be annotated with a list of all allowable substitutions.
Arrays
When mapping fields/properties of array type, the System.Xml.Serialization library always maps the field/property to a local element declaration. The name of the element is controlled using the [XmlArray] attribute. For each array type encountered, a named complex type is generated which contains a compositor marked minOccurs="0" and maxOccurs="unbounded". Inside that compositor resides another local element declaration that corresponds to the elements of the array. By default, the name and type of this inner local element declaration matches the CLR type of the array elements. You can control this mapping using the [XmlArrayItem] attribute. You should consider the following field declaration with no CLR attributes applied:
public Person [] people;
This would imply the following XML Schema type:
<xsd:complexType name="ArrayOfPerson" >
<xsd:sequence>
<xsd:element name="Person" type="target:Person"
minOccurs="0" maxOccurs="unbounded"
nillable="true" />
</xsd:sequence>
</xsd:complexType>
With this type in place, the people field would then map to the following local element declaration:
<xsd:element name="people" type="target:ArrayOfPerson" />
The [XmlArray] and [XmlArrayItem] attributes provide control over element names and nillability.
For arrays whose element type is polymorphic, you can typically rely on the xsi:type functionality of XML Schemas to be used. For more exotic uses, you can specify a poor-man's element substitution group using the [XmlArrayItem] attribute, as shown in Figure 22. In this example, the generated schema allows different element name/types in the innermost compositor (Figure 23).
Customizing XmlSerializer
The XmlSerializer type supports a variety of optimizations/customizations. In particular, you can provide a set of custom attributes (such as [XmlType], [XmlElement]) when initializing an XmlSerializer. These attributes override any attributes that may have already been applied to the CLR type. The primary utility of this feature is that it allows you to layer the XML serialization attributes onto preexisting classes that do not know about the XML serialization engine.
When deserializing, the XmlSerializer ignores unrecognized attributes, elements, character children, and processing instructions. For all but the last of these, that means that the deserializer will accept input documents that are not schema-valid. To allow developers to customize this behavior, the XmlSerializer type supports three events: UnknownAttribute, UnknownElement, and UnknownNode. UnknownAttribute and UnknownElement are fired whenever an attribute or element is encountered in the input document that is not mapped to a corresponding CLR field or property. The UnknownNode is fired whenever a child is encountered in the input document that is not mapped to a corresponding CLR field or property. This includes processing instructions and ignorable whitespace, neither of which affects schema validity. Figure 24 shows how to use the UnknownAttribute event.
Finally, during serialization, namespace declarations are emitted as they are needed. This means that if a particular namespace URI is used several times in the output document, there will be several redundant namespace declarations. This makes the output document correct albeit quite verbose. When serializing, you can pass a collection of namespace declarations to the Serialize method. This collection will be used to pre-declare namespaces at the root of the output document, suppressing the need for extra declarations in child elements.
.NET has much more Schema support than can fit into this column. Interesting bits left for future discussion include validation, getting at the PSVI via XmlValidatingReader, and schemas and data access.
Send questions and comments for Don to housews@microsoft.com. |
.gif)
.gif){kind=small}