---

June 2010

Volume 25 Number 06

Practical OData - Building Rich Internet Apps with the Open Data Protocol

By Shayne Burgess | June 2010

At PDC09 the Microsoft WCF Data Services team (formerly known as the ADO.NET Data Services team) first unveiled OData, the Open Data Protocol. The announcement was in a keynote on the second day of the conference, but that wasn’t where OData started. People familiar with ADO.NET Data Services have been using OData as the data transfer protocol for resource-based applications since ADO.NET Data Services became available in the Microsoft .NET Framework 3.5 SP1. In this article, I’ll explain how developers of Rich Internet Applications (RIAs) can use OData, and I’ll also show the benefits of doing so.

I’ll start by answering the No. 1 question I’ve been asked since the unveiling of OData in November: What is it? In very simple  terms, OData is a resource-based Web protocol for querying and updating data. OData defines operations on resources using HTTP verbs (PUT, POST, UPDATE and DELETE), and it identifies those resources using a standard URI syntax. Data is transferred over HTTP using the AtomPub or JSON standards. For AtomPub, the OData protocol defines some conventions on the standard to support the exchange of query and schema information. Visit odata.org for more information on OData.

The OData Ecosystem

In this article, I’ll introduce a few products, frameworks and Web Services that consume or produce OData feeds. The protocol defines the resources and methods that can be operated on and the operations (GET, PUT, POST, MERGE and DELETE, which correspond to read, create, replace, merge and delete) that can be performed on those resources.

In practice this means any client that can consume the OData protocol can operate over any of the producers. It’s not necessary to learn the programming model of a service to program against the service; it’s only necessary to choose the target language to program in.

If, for example, you’re a Silverlight developer who learns the OData library for that platform, you can program against any OData feed. Beyond the OData library for Silverlight you’ll find libraries for the Microsoft .NET Framework client, AJAX, Java, PHP and Objective-C, with more on the way. Also, Microsoft PowerPivot for Excel supports an OData feed as one of the options for data import to its in-memory analysis engine.

And just as clients capable of consuming the OData protocol can operate over any of the producers, a service or application created using OData can be consumed by any OData-enabled client. After creating a Web service that exposes relational data as an OData endpoint (or exposes the data in a SharePoint site, tables in Windows Azure or what have you), you can easily build a rich desktop client in the .NET Framework or a rich AJAX-based Web site that consumes the same data.

The long-term goal for OData is to have an OData client library for every major technology, programming language and platform so that every client app can consume the wealth of OData feeds. Combined, the producers and consumers of OData create an OData “ecosystem.”

What’s New in WCF Data Services?

WCF Data Services, a component of the .NET Framework, is a framework that offers a turnkey solution for creating OData Web services and includes a client library with which you can build clients that consume OData feeds. The WCF Data Services team recently released an update to the .NET Framework 3.5 SP1 that introduces a host of new features you’ll also find in the .NET Framework 4. This is the second version of the Data Services framework. Visit blogs.msdn.com/astoriateam/archive/2010/01/27/data-services-update-for-net-3-5-sp1-available-for-download.aspx, where you’ll find a description and a link for downloading.

The WCF Data Services framework is not just a protocol for RIA applications. It was also designed for high-scale service developers and has many features that appeal to them, such as server paging limits, HTTP caching support, stateless services, streaming support and a pluggable provider model. Let’s look at the new features that are generally of most interest to RIA developers.

One of the top feature wishes customers expressed after the initial release was the ability to request the number of entities in a set. The new “count” feature addresses that need with two parts. First, it lets you request only the count—that is, the number of values a query would return. Second, it adds a query option that tells the service to include a count of the total number of entities in a set when the query result is a partial set (for example, when server paging is enabled).

To enhance the experience when binding data from an OData service, a new type, DataServiceCollection, has been added to the WCF Data Services client library. It implements change tracking on the items it contains (through the use of the INotifyPropertyChanged and INotifyCollectionChanged interfaces). When it’s bound to a control—a DataGrid in Silverlight, for example—it will track the changes made to the objects and  to the collection itself. This new collection greatly simplifies the process of creating OData clients with an interface component.

Another frequently requested feature was the ability to project a subset of the properties of an entity returned in a query result. LINQ support has been added for this through the LINQ Select statement. This has two benefits: It reduces the size of the HTTP responses to queries, and it reduces the memory footprint of the client-side objects. This can be especially useful when you’re developing a client application against a service you don’t own and in which each entity may have many properties of no interest to the client. Later in this article, I’ll demonstrate working with a large, publicly available service that has many entities with numerous properties on each entity. Projections will be useful in the example because it includes only a few needed properties on one entity.

To help you understand the value of the OData ecosystem, we’ll create a Web application that lets visitors browse the site of my fictional real estate company, Contoso Ltd., to see the listings of the properties it manages.

Relational Data

The main data source for the Contoso.com Home Finder application is a SQL Server database that contains information about all of the properties the company is managing and all of the listings (current and previously sold) it has published for those properties.

Since the release of WCF Data Services and the ADO.NET Entity Framework in the .NET Framework 3.5 SP1, it has been easy to expose a relational database as an OData feed. All that’s needed is an Entity Framework model created over the relational data. An OData feed is HTTP-based, so you need a Web site or Web service to host the service.

To create the OData feed over the relational data, the first step is to create an ASP.NET Web application in Visual Studio 2010 to host the OData service. In Visual Studio, select File | New | Project | ASP.NET Web Application. This will create the skeleton of a Web service that can be used to host the OData feed.

After the Web service is created and configured, we’ll create the Entity Framework data model that the OData feed will expose. Visual Studio makes this easy using the Add New Item wizard, which lets you auto-generate a model from an existing database. Figure 1 shows a simple data model created using the Add New Item wizard over the SQL Server data containing the properties and listings managed by Contoso.

Figure 1 The Entity Framework Data Model for the Relational Data

Now let’s create a WCF Data Service that exposes this data model as an OData feed. Visual Studio also makes this simple with the WCF Data Service option in the Add New Item wizard. When you select this option, Visual Studio provides a code file (in this example the file is called Listings.svc.cs) that’s used to configure the Data Service.

The code in Figure 2 demonstrates how to define a WCF Data Service. The Listings class is the service class that’s exposing the Data Service, and it implements the generic DataService<T>. The type used to define the DataService<T> in Figure 2 is the ListingsEntities type, which is the Entity Framework context created in Figure 1. Because this class will accept an Entity Framework context, this is a quick and easy way to get a WCF Data Service that exposes relational data up and running. The DataService class isn’t restricted to just working over Entity Framework contexts, however; 
the class will accept any collection of CLR objects that implements the IQueryable interface. In the .NET Framework 4, a new custom provider model for WCF Data Services has been added that allows a service to be created over almost any data source.

Figure 2 Defining the WCF Data Service

// The ListingsEntities is the Entity Framework Context that the Service exposes
public class Listings : DataService< ListingsEntities >
{
  public static void InitializeService(DataServiceConfiguration config)
  {
    // These lines set the access rights to "Read Only" for both entity sets
    config.SetEntitySetAccessRule("Listings", EntitySetRights.AllRead);
    config.SetEntitySetAccessRule("Properties", EntitySetRights.AllRead);
    // There are currently no service operations in the service
    config.SetServiceOperationAccessRule("MyServiceOperation",
      ServiceOperationRights.All);
    config.DataServiceBehavior.MaxProtocolVersion = 
      DataServiceProtocolVersion.V2;
  }
}

Let’s look a little closer at what else the InitalizeService method in Figure 2 is doing. The method is calling the SetEntitySetAccessRule for both of the entity sets the service will expose and setting the access rights to AllRead. This tells the service to make both entity sets fully readable but not allow any inserts, updates or deletes. This is a great way to control access to the service. WCF Data Services also support methods called Query Interceptors that allow the service author to configure finer-grained access control for the service on a per-entity-set basis. Set the Listings.svc file as the project start page and run the project. A browser window will open and display the service document, as shown in Figure 3.

Figure 3 Service Document for the SharePoint Site

OData URI Conventions

The service document lists the entity sets that are exposed by the service. Remember, you can access the resources in this service using the powerful URI syntax defined as an optional part of the OData protocol. Let’s take a quick look at the URI syntax for this service. To access a feed for each entity set, you append the name of the entity set to the base URI for the service; for example, https://myhost/Listings.svc/Properties would address the set of entities in the Properties entity set.

It’s also possible to address a particular entity individually using its key value; the URI https://myhost/ Listings.svc/Properties(0) would address the property with ID = 0. You can address a relationship from this entity to another entity or set of entities by appending the name of the relationship to the end of the URI, so https://myhost/ Listings.svc/Properties(0)/Listings would access the set of listings associated with the property entity with ID = 0. Using this syntax, it’s possible to navigate through many levels of relationships.

The URI syntax also defines a number of query options that can be appended to a URI to modify the base query in some way, and each query option is defined as a name/value pair. For example, by appending the query option $top=10, you restrict the query to only the first 10 entries in the result. Figure 4 lists all of the query options available in the URI syntax.

Figure 4 OData Query Options

$top=n	Restricts the query to the first n entities.
$skip=n	Skips the first n entities in the set.
$inlinecount=allpages	Includes the count of all entities of the set in the result.
$filter=<expression>	An expression can be supplied to restrict the results returned by the query (example: $filter=Status eq ‘Available’ restricts the results to entities that have a Status property with the value “Available”).
$orderby=<expression>	Orders the results by a set of properties of the entity
$select=<expression>	Specifies a subset of the properties of the entity to be returned.
$format	Specifies the format of the feed to be returned (ATOM or JSON). This option is not supported in WCF Data Services.

Exposing Data from SharePoint

In the preceding section I showed you how to expose the data stored in my relational database, the property and listing information for the real estate Web site. Let’s say I also have information about the real estate agents who are selling the properties, but that data is stored in a SharePoint site. Microsoft SharePoint 2010 has the ability to expose all lists and documents within those lists as an OData feed. This is great for the real estate site because it means the agent information that company employees have entered is available as an OData feed that can be included in the listings application I’m building. The users who have processes using the SharePoint interface for entering and updating this data don’t have to change their workflow to suit my application. The data entered into the company SharePoint site is available in real time to the Listings application that’s being created.

Figure 5 shows the simple SharePoint portal the real estate agents use to record and update their contact information.

Figure 5 SharePoint Site for Agent Information

When the ADO.NET Data Services Update for the .NET Framework 3.5 SP1 is installed on the SharePoint system, a new HTTP endpoint becomes available for each site that exposes the list data as an OData feed. Because an OData feed is accessed using HTTP, it can be examined by using just Internet Explorer. Figure 6 shows the feed for the agents list in SharePoint.

Figure 6 Agents Feed from the SharePoint Agent Service

Consuming Reference Data from OGDI

By default, an OData feed will return an ATOM representation for the feed, and when accessed from a Web browser the result will be an ATOM feed. If the accept header of the request is changed to “application/json,” the result will be the same data as a JSON feed.

The feed in Figure 6 starts with a <feed> element that represents a set of entities. Contained within each feed is a set of <entry> elements, each of which represents a single entity in the feed (the first three entry elements are collapsed to make the whole feed visible in one screen).

In this example, the entity has a concurrency token defined on it; as a result, each entity in the feed has an etag property on it. The etag is the token used by the data service to enforce a concurrency check when a change is made to the requested entity. Each entity, formatted using an <entry> tag, consists of a set of links that contain both the link to be used when editing the entity and the entity’s relationships. Each relationship link points either to another entity or to a set of entities (these are called reference and navigation properties, respectively). Each <entry> element also 
includes an <m:properties> element that contains the primitive and complex type properties for the entity; the property values consist of the name of the property on the entity and the value for that property.

The Open Government Data Initiative (OGDI) is a service built on the Microsoft Windows Azure that makes it easier for government agencies to publish a wide variety of public data. The OGDI project provides a starter kit that can be used by government agencies to expose their data. For example, the city of Edmonton has adopted the starter kit to expose its government data, and a service at ogdisdk.cloudapp.net has a data set with a variety of data about the Washington, D.C., area. Another example is the Microsoft Codename “Dallas” project that aims to make it simple for anyone with a data set to expose the data as a service to the Web. This project is also built on the Windows Azure and exposes data using OData. These are examples of high-scale services that expose large reference data sets that can be further consumed by Web applications. As I will show, when these services expose their data using OData, it’s simple to consume that data from a variety of applications.

As explained, the OGDI Web site features publicly available data about the Washington, D.C., area. Contoso’s real estate application is used for browsing listings in that area, and it would be helpful for users to have available some of this reference data about the area around a particular property when viewing it. When I create the client for the sample application, I’ll demonstrate how to include the OData feed from the OGDI Web site as one of the sources of data for the application.

Other OData Producers

So far I’ve shown examples of consuming data from SQL Server, SharePoint and a generic OData service on the Web, but more options exist. The cloud-based Windows Azure has a table service that exposes data stored in Windows Azure tables, and the API for this is built using OData. As mentioned, the Microsoft Dallas project is a data marketplace for finding and querying data exposed by the Dallas service, and this service exposes its data using the OData protocol. OData producers aren’t just limited to Microsoft products either; IBM recently announced that its WebSphere eXtreme Scale 7.0 product now supports the OData protocol.

Silverlight Client

Contoso’s real estate finder application now has an ASP.NET Web service that exposes the relational data in SQL Server about the real estate listings and properties managed by the company; a SharePoint site that’s being used to manage the data about company agents; and a government Web service that exposes data about the region around the properties the company is advertising. I want to put all of these sources together into one Silverlight application that can work with this data in a meaningful way.

In Silverlight 3, a WCF Data Services client library is included in the Silverlight SDK that makes it simple for Silverlight applications to communicate with a service that’s OData-enabled. To do this, in Visual Studio from a Silverlight project, right-click the project and select Add Service Reference. This walks you through the process of creating a service reference. The main input to a service reference is the URI of the service that’s being referenced from the Silverlight application. Figure 7 shows an example of adding a service reference to the OGDI sample service.

Figure 7 Add Service Reference for the OGDI Sample Service

The service reference wizard creates a client-side context class that’s used to interact with the data service. The client context abstracts the details of working with HTTP and URIs away from the client programming model and allows the client developer to think only about C# classes and XAML. The client context also includes a LINQ provider implementation and, as a result, LINQ queries on the proxy are supported. The Add Service Reference wizard will also generate a set of client proxy classes that mirror the types that are exposed by the referenced service. After creating the OGDI service reference, I will also create a service reference to both the SharePoint and Listings services I created. This code shows how to create the contexts that are used to interact with the three OData services:

// OGDI service context
OGDIService.dcDataService OGDIService = 
  new dcDataService(new Uri(OGDIServiceURI));
// SharePoint service context
AgentsReference.AgentsDataContext agentsService = 
  new AgentsReference.AgentsDataContext(new Uri(sharepointServiceURI));
// Listings Service context
ListingReference.ListingsEntities listingsService =
  new ListingReference.ListingsEntities(new Uri(listingsServiceURI));

Figure 8 shows the outline of the Silverlight real estate Home Finder application. The application will be hosted in SharePoint so that it’s easily available for my existing users who are used to working in the SharePoint environment.

Figure 8 The Contoso Home Finder

Figure 9 contains the code for querying the listings service and binding the result to the grid at the top of the Home Finder Silverlight application.

Figure 9 Creating the Client Proxy Contexts

private void getListings()
{
  DataServiceCollection<Listing> listings = new 
    DataServiceCollection<Listing>();
  listingsGrid.ItemsSource = listings;
  
  var query = from listing in
              listingsService.Listings.Expand("Property")
              select listing;
  listings.LoadAsync(query);
}

The code in Figure 9 creates a DataServiceCollection that’s a tracked collection and binds the collection to the ItemsSource property of the main listings grid. Because this collection implements change tracking, any changes made to the items in the grid will automatically be reflected on the entities in the listings collection. The changes in the grid can be persisted to the service by calling the BeginSaveChanges method on the context for the listings service.

In Silverlight, all network calls are made asynchronously, so executing any operations against a service using the WCF Data Services client library involves making the initial call to the operation and then writing a separate callback method that’s passed to the method to handle the result of the asynchronous call. To improve this asynchronous experience, a method was added to the DataServiceCollection class, LoadAsync, which does all the work of handling the asynchronous callback function and loading the results into the collection.

In the Figure 9 code, the collection is bound to a grid before the LoadAsync call is made, and the values won’t be loaded into the collection until after the asynchronous call completes. The collection will raise collection-changed events when the results are returned from the service, and the grid will catch those events and display the results when the asynchronous call completes.

When a listing is selected from the data grid, the SharePoint site needs to be queried to get the information about the agent managing that listing. In this application architecture, a second query is needed because the data sources for the listing type and the agent type are separate and there’s no explicit relationship between the two (if you are a person who thinks in terms of models, this example involves two completely separate models, and the relationship between the models is an artificial one created and enforced by the client).

Figure 10 shows how to query the SharePoint service for the agent entity, given the name of the agent. A similar code sequence is used to query the OGDI data for the neighborhood statistics in the chart at the bottom of the Home Finder page. The code up to this point demonstrates only the query capabilities of the Silverlight client, but the client isn’t limited to queries only; it has rich capabilities in writing back changes to the service from the client.

Figure 10 Executing an Asynchronous Query

private void GetAgentData(string agentName)
{
    var query = agentsService.Agents.Where(a => a.FullName == agentName) as 
        DataServiceQuery;
    query.BeginExecute(
        AgentQueryCallBack, query);
}
private void AgentQueryCallBack(IAsyncResult result)
{
    Dispatcher.BeginInvoke(() =>
    {
        var queryResult = result.AsyncState as DataServiceQuery<AgentsItem>;
        if (queryResult != null)
        {
            var agents = queryResult.EndExecute(result);
            this.grdAgent.DataContext = agents.First();
        }
    });
}

OData in PowerPivot

PowerPivot is a new in-memory business intelligence tool that’s delivered as an add-in for Microsoft Excel 2010—visit powerpivot.com for more information. The tool provides support for importing large data sets from a data source and doing complex data analysis and reporting. PowerPivot can import data from a number of different data sources, including directly from an OData feed. PowerPivot’s From Data Feeds option (shown in Figure 11), accepts an OData service endpoint as the location of the feed to import.

Figure 11 PowerPivot Imports from an OData Feed

Figure 12 shows a chart made from the summary of crime statistics in the OGDI’s Washington, D.C., data feed.

Figure 12 PowerPivot Chart from OData Feed

The chart in Figure 12, made using the same data set as the real estate application in the previous example, shows a summary of all data for each district. I encourage you to download PowerPivot for Excel 2010 and import data from the OGDI site at https://ogdisdk.cloudapp.net/, and see for yourself how quickly you’ll be doing rich data analysis over this data.

The Open Data Protocol Visualizer

The OGDI data service is essentially a “black box” to an external developer who creates an application that consumes the data exposed by the service. Thankfully, the OGDI service exposes its data using the OData protocol, so there’s no need to know anything about the internal details of the service to interact with it. The programming model for the service is the OData protocol. The service endpoint describes the shape of the data and, as I showed you in the previous section, that’s all you need to interact with the service. However, it’s often useful to view the shape of the data in the service and get a better understanding of the relationships between parts of the service. The Open Data Protocol Visualizer was created for just this purpose. It’s available from the Tools | Extension Manager menu item in Visual Studio 2010. Figure 13 shows two views from the visualizer that display the structure of the OGDI service.

The top view in Figure 13 shows the entire service; the bottom view is zoomed in to show only four of the boxes in the display. The visualizer represents entity sets as boxes, and relationships between entities as lines connecting the boxes. It’s clear from the top view in Figure 13 that the OGDI service is entirely flat and doesn’t contain any relationships at all, as there are no connecting lines between any boxes. This is only a characteristics of the OGDI service and isn’t typical of most OData services. The bottom view shows a close-up of four of the entity sets in the service. Just from examining the view, you can determine that the service exposes data about fire stations, elementary school attendance, dialysis clinics and government locations, as well as the properties and keys for each of those types.

Figure 13 Open Data Visualizer Views of the OGDI Sample Service

Learning More

This article introduces the Open Data Protocol and the ecosystem that has been built around it, including the WCF Data Services Framework. For more information, visit the Open Data Protocol Web site at odata.org. To learn more about WCF Data Services, see msdn.microsoft.com/data/bb931106 or the WCF Data Services blog at blogs.msdn.com/astoriateam.

---

Shayne Burgess is a program manager in the Data and Modeling Group at Microsoft, working specifically on WCF Data Services and the Open Data Protocol. Burgess regularly blogs on the WCF Data Services team blog at blogs.msdn.com/b/astoriateam.

Thanks to the following technical experts for reviewing this article: Elisa Flasko and Mike Flasko