Wordament, part II

Developing a cross-platform app

By: Sidney Higa and Walter Poupore

 

C# | Cross Platform | Azure | Xamarin

In Wordament part I, Jason Cahill and John Thornton revealed the architecture of their hit game Wordament. In this segment, we’ll learn how they efficiently coded Wordament for release to multiple platforms, beyond the initial release that was for Windows Phone only.

We’ll look at Xamarin and partial classes, which allowed Jason and John to write all of their code in one programming language while keeping a significant amount of code common across their Android, iOS and Windows Phone releases.

We’ll also see how they use Azure services as the comprehensive backend for Wordament, and learn about other tools they consider invaluable for cross-platform development.

Wordament is played on many kinds of devices from phones, tablets, PCs, to Xbox. Each device presents a monumental challenge to game developers because each device has a different operating system and APIs. The team used Xamarin and partial classes to keep the client code base manageable.

Building a cross-platform, cloud-connected game requires a set of tools and technologies to support game UX and behavior, data flows, and data storage.

Wordament is built entirely in C#, using Visual Studio as the IDE, including all client apps as well as the services.

Using C# has great benefits; it’s:

1. A modern general-purpose programming language
2. Backed with the extensive functionality in .NET libraries
3. Easy to edit using the Visual Studio editor (IntelliSense, auto-completion, compiler checks, etc.)

According to Wordament’s creators, the biggest advantage is that developers and testers all use the same language—the development process is optimized when every part is written in the same language. Debugging is easier. Communication is smoother. As John says, "Everything is just C# all the time, and everything goes through Xamarin. That makes it super simple for us to add features to any platform we want to go to, and to release on new platforms. By writing everything in C#, we definitely created a world where it is very fast for us to get to other platforms."

As John mentions, they’re using Xamarin. This allows the Wordament team to leverage C# for Android and iOS development, making this a good time to talk about cross-platform development.

Developers and testers all use the same language

Wordament is supported on a variety of platforms. About 80% or more of the code base is common to all platforms.

Given the number of different devices, there’s a need to have good code management capabilities for the platform-specific portions.

The Wordament team uses partial classes to minimize #if directives, and uses Xamarin for cross-platform C# compilers and .NET runtimes to keep the code base manageable.

The development team worked on the Windows and Windows Phone releases first, building the whole app in Visual Studio. Then they continued to use Visual Studio to write the platform-specific code for Android and iOS using Xamarin’s wrappers of Android and iOS native APIs. Xamarin is constructed to allow you to code in C#, as shown in image, to produce executables for the native OS calls. Keeping it in one code environment again reduces the friction for the coding team. Developers do not have to learn new programming languages, which cuts down on "porting bugs" and logic errors. They can concentrate on the code, and fix or implement features across all platforms in one space.

Another way to reduce friction is to use C# partial classes. Partial classes enable you to split up a class into multiple files. This means you can separate, for instance, the platform-dependent UI drawing code from the platform-independent UI layout and logic code.

For Wordament, partial classes are used to segregate “common” code and platform-specific code. For example, consider a UI button in a Wordament client.

Portions of UI button functionality may be common across all client platforms, but some would be specific to the native OS running on the underlying hardware.

Partial classes are a feature of C#, and you can use them to create code files that separate platform-specific code from common code. Methods in the common file call into platform specific code to create the proper executables.

As shown in the diagram, a partial class named Rectangle contains the common code needed for to draw a rectangle. It's in a file named rectangle.cs. A second file named rectangle_android.cs contains the code specific to Android devices for constructing a rectangle. 

The rectangle.cs file contains the constructor that calls the CreatePlatformControl method on every device. It also holds the Color property, which calls a common method named UpdatePlatformColor. The rectangle_android.cs file contains a definition of both methods specific to the Android platform. Similarly, the rectangle_win.cs file shows the same methods with the code for a Windows Phone client. At compile time, the correct file is used to compile for each platform.

Another benefit this is easier debugging, as the code is partitioned into separate files. Unless the common code is being revamped, debugging will be isolated to the device-specific file, minimizing entry points.

In the Wordament solution layout, there's a separate project per client.

If only the Windows Phone 8 code needs to be modified, it can be done without requiring changes to the iOS code. Additionally, if a breaking change to platform-specific code is introduced into the common code, the next time that a platform-specific project is compiled or opened, the developer will know about it.

More information about using Xamarin for mobile services is available in this video. To learn more about partial classes, see Partial Classes and Methods on MSDN.

The backend story is conceptually much simpler. The Visitor Center, Game Room, Aggregator, and Leaderboad Orchestrator (LBO) are all functioning purely with data. As such, there’s no need to take into account the form factors of the individual client apps.

Handling and storing data is a vital part of any app story. In the case of Wordament, data also has the critical function of being the central "currency" among the various services. In the loosely coupled world of web services, the time it takes for any service to finish processing is unpredictable. However the data is constantly moving between the pieces, like coins in an economy, passing from one component to another, used over and over again. For example, the results of a single game are stored in Azure storage by one process. Another process takes the results and pulls out the leaderboard scores. Another process calculates "Best of" statistics. Still another process updates the histories of each player with the new results. Each of these processes can work independently of each other, but they are all processing the same data set.

The primary keys

There are two essential kinds of data, and therefore two important primary keys. The first is the game ID itself. All players play the same game for a given puzzle language, which has a single unique ID. The other is the player ID. Each player’s history is tracked—each game and its result. Over a span of time, the player’s averages can be calculated.

As shown in the data workflow illustration, data is handled by only two components: the Game Room and the Leaderboard Orchestrator. The GR creates files of the game results that are then stored in Azure blob storage. The LBO reads the game results and then processes it for insertion into Azure SQL databases. There is one bonus in this configuration: the handling of data is isolated from the web-facing servers. This mitigates the harm that could be inflicted if the servers are hacked. Even if hackers control the servers, they can only send or retrieve data—but cannot change it directly.

Eventual consistency

Wordament was built with resiliency in mind—such as the abundant use of error handling in their code. Another strategy is the use of eventual consistency. That is a tolerance of data sets that are not complete, but will eventually be so. For example, when waiting for clients to send game results in, the Game Room waits for 17 seconds and uses the current data to calculate leaderboard scores. The final set of scores is not complete until after the stragglers come in (beyond the 17 seconds). However, there is no use in waiting for stragglers just to show the mostly correct leaderboard. This same strategy is used throughout the data handling in Wordament.

The game relies heavily on Azure storage partly because it is so economical. The bulk of the data is stored as blobs in Azure storage. The information stored there is composed of only three file types:

1. Player information (name, history) in table storage
2. Game results, stored as blobs
3. Persistent leader results, also stored as blobs

Azure SQL Database

Wordament also uses Azure SQL Database for storing a week’s worth of data. One advantage of using SQL is that stored procedures can be quickly run against the data. Stored procedures are inherently safer for any data operations, as they are well constructed and parameterized. The team also uses the results of stored procedures to create the persistent leader results. To keep the round-trips to the database at a minimum, the team uses table-valued parameters. One of the benefits of table-valued parameters is they allow multiple rows of data to be written to the database in a single transaction (rather than inserted one row at a time). For additional tips about performance and scalability with SQL Database, see Batching Techniques for SQL Database Applications in Azure.

The game results is a CSV file that is stored as a blob in Azure Storage. There are two fields that every row contains: the game ID and the Wordament User ID (WUID). The game ID specifies the actual game played. The WUID (Wordament User ID) points to the player. Following that, the file lists the actual game statistics for each player—score, best word, as well as any other data that the developers want to include in the results file.

By using a CSV, John and Jason have developed an elegant method of ensuring compatibility with future versions, while allowing functionality to grow. In short, a CSV allows them to simply add data to any file by adding commas.

Older versions of their clients will read only as many fields as they expect—but will disregard any data that follows. So while an older client may not get the full value of the latest data file, it also won’t break.

As the developers put it, they designed their data schema to "extend down and to the right." They can add as many rows as they like and add as many fields as they like, without breaking backward compatibility.

The only use of table storage comes in holding player information. Each row in a table is an entity, and each entity can have up to 255 properties. The table is used because the player info might not be found in the cache, so a search must be made. Tables are optimized for queries.

The player tables also hold these records:

• best word
• best score
• average score

Persistent leader results are the collection of current leaders in the Wordament universe—the top scorers and their scores. The data is very similar to that of the game results file. The results are built after every game, and the latest version is placed in a special, well-known folder that can be accessed by the Visitor Center for display there.

Wordament uses Azure SQL databases to calculate the statistics, and to generate the persistent leaderboard files. This happens after every game—a huge task. It takes the results of every game played in every language. It also uses the results of the past week to calculate the "best of week" leaders. All of the calculation is done by populating Azure SQL with the latest game data, along with the data from the past week, and running a multitude of stored procedures.

To keep the SQL Azure database as lean as possible, the Wordament team is "hyper aggressive about truncating" unneeded data rows. Only the minimum amount of data is stored to satisfy queries such as "best of week," "Best of hour," "Best of right now," etc. This limits the amount of storage used, but takes full advantage of the benefits of a relational store and the ability to run stored procedures against the data.

For example, the Wordament database has a table to support real-time querying of "best of the hour" results. Data is maintained on a rolling hour basis. As data rows become older than the current hour, they are removed from the table. The truncation is handled by a stored procedure. Similarly, other tables handle the "best of the day" and "best of the week" results. And again, stored procedures run to truncate older rows when they’re no longer needed.

Keeping the tables free of unneeded rows minimizes Wordament’s SQL Azure costs, and also helps optimize database performance for the leaderboard queries.

The Visitor Center doesn’t query SQL Azure directly when it needs to serve leaderboard results.

Instead, stored procedures are run on SQL Azure to generate the CSV files which are stored in blob storage. The various Wordament components understand the underlying storage schema, so for example the VC can pull the data as needed. A Last Known Good (LKG) version of the latest available results is always available to the components. The file name for LKG results doesn’t change when the data is updated, the various components just use the same file.

One week includes a fixed number of games. The only variable is the number of players in one week.

That’s what Jason and John say: log everything—including all requests, game results, unhandled exceptions, and use bulk logging for everything else. "We log everything in our service," says John. "We overlog basically. We log everything that happens, then we can always go back and search through that to find things that are problems or patterns."

Note that all this logging occurs on the backend components—such as the Game Room and LBO. For the unhandled exceptions and criticals, the entire call stack is logged, which can then be used in Visual Studio to go directly to the line of code that failed. PowerShell scripts are used to download the logs and also to facilitate querying of the logs.

One of the prime benefits of using Azure storage is its low cost. At this point in time, one gigabyte costs only $.07 a month. Wordament’s storage costs only $45 a month—about 643 GB.

When logging so much information, it’s important to figure out what is important. There are two systems at work here. The first is Windows Azure Diagnostics. That system automatically downloads the logs from each worker or web role to Azure storage. That kind of data is useful when checking for patterns or possible bottlenecks. But when the system is in trouble, you need an immediate alert.

For this, Jason and John invented their own logging system that prioritizes errors. In a rough ranking from one to ten (ten being deadly), errors are checked for severity. The most critical errors are unhandled exceptions. When one occurs, the team is notified via email, and the problem is tracked down as soon as possible.

The team is adamant about one thing: they debug by logging. They do not even enable the ability to Remote Desktop into any instance. That precludes the chance of debugging the role instances while they are running. If they cannot solve the problem by examining the log, then they refine their logging system.

Log everything—including all requests, game results, and unhandled exceptions

Wordament uses "a ridiculous amount of error handling" in the client. That is a best practice for any cloud service. There are latencies galore in the mobile world.

Every tunnel or building can cause a dropout for a second or minutes. While not in the scope of this article, suffice to say that you must use C#’s try-catch methodology to trap anything that will interfere with the game.

We’ve covered how an evolving app can be managed even as it continues to run, with thousands of users simultaneously playing—and never noticing a change.

Some of the general development lessons from the Wordament team are:

• Data is the one constant in any backend, and developing a rational schema keeps everything synchronized.

• Use “eventual consistency”—when possible—to achieve data integrity.

• CSV files are a simple way to extend your data types while maintaining backwards compatibility.

• Use Azure SQL Database to handle your data-intensive processes; use blob storage for archiving; use Azure table storage to store player data that needs to be occasionally updated.

• Log the errors and watch for patterns that can be mitigated or eliminated.

When it comes to the cross-platform development of Wordament, here's the key lessons shared by Jason and John:

• C# can be used with Xamarin to develop client apps for Android and iOS.

• Using one programming language reduces the development and debugging time of the app, and minimizes communication problems when a team grows and includes members in different time zones.

• Partial classes—a feature of C#—lets you segregate your code simply.

• Azure provides a comprehensive set of reliable and scalable services for the backend of cross-platform apps, letting you focus on your app development instead of being sidetracked by IT tasks.

Part III—Wordament: Running a business

In the next part of this series, the Wordament team reveals more tips for keeping a responsive and performant massively multi-player game running 24/7.