January 2014

Volume 29 Number 1

Windows 8.1 : Create Modern Microfiche with the Chronicling America API

Tim Kulp

This article is from MSDN Magazine's special coverage of application development in the government sector.
Read more articles like this in the Special Government Issue of MSDN Magazine.

Today, the newspaper is being replaced by other news sources on the Web. At one time, though, newspapers were the primary source of information about current events. Through the Chronicling America project (chroniclingamerica.loc.gov), the U.S. Library of Congress and various institutions have been working to make the rich history of U.S. newspapers available on the Web via a standardized API. This API lets developers leverage media from the Chronicling America project to build research apps, news apps with historical context or apps for attaching content to family genealogy. In this article, I’ll introduce the Chronicling America API and build a Windows Store app that lets users research and view digitalized newspapers.

Overview of the Chronicling America API

Chronicling America has two core components to its API: the OpenSearch protocol and standardized URLs to navigate content. For each publication, the URL follows a standard format that you can modify to provide different data. As an example, retrieving a specific page requires the following URL structure: https://chroniclingamerica.loc.gov/lccn/sn83030214/1913-08-12/ed-1/seq-1/. This format breaks down to: requesting a specific publication (/lccn/sn83030214/), providing the date and identifier of the issue and edition (1913-08-12/ed-1), and providing the page number (seq-1/). If I want to request the second page, I just update seq-1 to seq-2. Throughout this article, I’ll use the standardized URL to make navigation simple and fast without having to look up objects with many one-off requests to the server.

You can use the URL to retrieve specific content, and the OpenSearch protocol to search titles and pages. Searches against titles and pages use different URL structures. Title search structures can be found at 1.usa.gov/1a0GOpF, and page search structures can be found at 1.usa.gov/188T1HB. These XML files provide all the output formats and parameters that can be used in Chronicling America. I’ll focus on page searches in this article to have the most properties against which to search.

Building a Microfiche App

If you aren’t old enough to remember microfiche, it was a clear piece of film on which miniature images were printed. These images were barely readable by the naked eye, but by putting the microfiche sheet in a special projector, you could gain access to hundreds of images (or more). It was a great mass-storage medium for images and helped users quickly move between documents by using the projector’s control wheel.

Microfiche is my inspiration for a sample app that displays newspapers from Chronicling America. The app starts with a list of publications displayed in a GridView. Users can search this content (an enhancement from microfiche) and examine an issue, much like using an eReader. The Library of Congress has already built a nice Web application with robust JavaScript capabilities. You can learn from this site and translate JavaScript from the Web to build the Windows Store app.

I built my Chronicling America app in JavaScript for Windows 8.1. To begin, create a Navigation app from the JavaScript Windows Store app templates in Visual Studio. Use the preexisting code in the template (for example, the navigation code in navigator.js and the app activation event handler in default.js) so you don’t have to manually build all your app’s infrastructure.

100 Years Ago Today …

The Chronicling America homepage displays newspaper front pages for today’s date 100 years ago. This is a fun place to start my app and immediately introduce people to the content. To begin, I need to connect to Chronicling America and pull data. The app performs many functions around its data, so in the js folder, I create a new JavaScript file named data.js, where all data interactions will occur. Think of this as my data layer.

I add the following code to data.js to create a self-contained function, because I don’t want the code in data.js to leak out into the global namespace:

(function () {
  "use strict";
  // Code goes here
})();

This block makes anything inside it private so I don’t encounter any variable-naming conflicts with files that use data.js. The comment, “Code goes here” is replaced with the code in Figure 1.

Figure 1 Initial Code for Data.js

var imageList = new WinJS.Binding.List(); // The image list for binding
var pagesList = new WinJS.Binding.List(); // List of pages for a single issue
var urlPrefix =   "https://chroniclingamerica.loc.gov"; // Used to prefix all URLs
var httpClient =   new Windows.Web.Http.HttpClient(); // Default HTTP client to use
function getInitialPages() {
  // Get the date and set it 100 years ago
  var hundredYearsAgo = new Date();
  hundredYearsAgo.setYear(hundredYearsAgo.getYear() - 100);
  var uri =
    new Windows.Foundation.Uri(urlPrefix + "/frontpages/" +
    hundredYearsAgo.getFullYear() + "-" + hundredYearsAgo.getMonth() + "-" +
    hundredYearsAgo.getDate() + ".json");
  httpClient.getStringAsync(uri)
    .then(
    function c(result) {
      var images = JSON.parse(result);
      images.forEach(function (item) {
        item.medium_url = urlPrefix + item.medium_url;
        item.thumbnail_url = urlPrefix + item.thumbnail_url;
        item.url = urlPrefix + item.url;
        imageList.push(item);
    });
  })
}
// Get function goes here
WinJS.Namespace.define("Data", {
  GetInitialPage: getInitialPages,
  ImageList: imageList,
  PageList: pagesList
});

Figure 1 starts with basic declarations that hold the data from the Chronicling America API calls. I set up some housekeeping variables, such as a URL prefix to avoid typing (and inevitably mistyping) the base Chronicling America URL, as well as an HttpClient object to be shared across functions.

The first function in data.js calls the initial pages from 100 years ago. This function starts with a simple calculation to get today’s date and then subtracts 100 years. Using the standardized URLs from Chronicling America, I set up a request to the frontpages directory of the API to load all the front pages from every issue matching the provided date. Notice the URL ends with .json to denote that I want JSON data back from the server.

The Chronicling America API lets you set this value to request different data types, such as XML or HTML. Using the httpClient.get­StringAsync function, the app requests the JSON data from the API and parses the returned string value into a JSON object. Because I’m calling getStringAsync, I don’t know what type of data is coming back from the server. My app expects JSON, so to verify the data the API returns can be converted to JSON, I use JSON.parse. This returns an array of JSON objects in this case. If the returned string can’t be converted to JSON, the code raises an error. Finally, the code updates the URLs in the item to append the prefix (because Chronicling America provides the URLs here as relative URLs, not absolute) to ensure my app can connect to the images.

One note on using Windows.Web.Http.HttpClient: Always plan on failure. The server your code is trying to reach could be down or unavailable, for example. Error handling has been omitted here for brevity, but I’ve included it in the online sample code.

At the end of the code in Figure 1, I added WinJS.Namespace to register the contents of data.js to be available to other applications. This namespace capability converts the private objects in data.js to public, accessible by other code. Using WinJS.Namespace.define informs an app that the defined object and members are available to other parts of the code. In essence, this is where you make Data and its members accessible with IntelliSense on your pages.

Binding to the Home Screen With the Data object defined, you need to connect the HTML page to display content from the Chronicling America site. Start by opening the default.html page and add a script reference to data.js.

This step makes data.js available to all the pages in the app. Replace the code in the Main section of home.html with the following:

<div id="hundresYearsListView" class="itemslist win-selectionstylefilled"
  aria-label="List of Publications"
  data-win-control="WinJS.UI.ListView" data-win-options=
    "{ selectionMode: 'none', layout: { type:  WinJS.UI.GridLayout } }"></div>

This code defines a ListView control that displays search results from the Chronicling America API. The ListView control uses a template with an image of the newspaper page and the place of publication. By using a WinJS.Binding.Template control, you describe what should appear in the ListView. Templates let your design team (or marketing team, if you don’t have a design team) work directly with HTML without worrying about data binding. Add the following directly below the <body> tag:

<div class="itemtemplate" data-win-control="WinJS.Binding.Template">
  <div class="item">
    <img class="item-image" src="#" data-win-bind=
      "src: thumbnail_url; alt: label" />
    <div class="item-description">
    <h3 class="item-title" data-win-bind="textContent: label"></h3>
    <h4>(<span data-win-bind="textContent: pages"></span> pp.)</h4>
    <h4 class="item-subtitle win-type-ellipsis" data-win-bind="textContent:
      place_of_publication"></h4>
    </div>
  </div>
</div>

Determining which property to bind to which field is handled by the data-win-bind attribute, which Windows uses to determine what to display in specific attributes during binding. The dataset attribute is a new attribute in HTML5 that lets you build custom attributes for your needs. Windows Store apps make extensive use of the data- attribute, but you can build your own attributes, such as data-publication in this example, which holds the name of the newspaper. This attribute can be searched by using the document.querySelector function, providing easy access to all the elements with this attribute.

Connecting Data to the ListView To put data into the ListView, open the home.js file (/pages/home/home.js) and add the code shown in Figure 2 to the ready function for the Home page control.

Figure 2 Code Added to the Ready Function

ready: function (element, options) {
  // Load the data
  Data.GetInitialPage();
  // Configure the list view
  var hundredYearListView =
     document.getElementById("hundresYearsListView").winControl;
  hundredYearListView.itemDataSource = Data.ImageList.dataSource;
  hundredYearListView.itemTemplate = document.querySelector(".itemtemplate");
  hundredYearListView.oniteminvoked = this._itemInvoked;
  // Process the bindings
    WinJS.Binding.processAll();
},
_itemInvoked: function (args) {
  args.detail.itemPromise.done(function itemInvoked(item) {
  var itemSelected = Data.ImageList.getAt(args.detail.itemIndex);
  WinJS.Navigation.navigate(
    "/pages/details/details.html", { selectedItem: itemSelected });
  });
}

The code in Figure 2 begins the process of loading data from the API by calling the Data.GetInitialPage function. Then I set up the ListView to pick up data from the ImageList (which is populated by the GetInitialPage function) and associate the template control to the ListView control. To wrap up the ListView control’s capabilities, I add an event handler for ItemInvoked so when a user clicks a newspaper image, something happens. The ready function ends with a call to WinJS.Binding.processAll to update the UI based on the modifications that happened during the ready function, such as data binding.

Next, _itemInvoked is defined to take a selected item and pass it on to the details page for further action. This is a basic operation of just passing the data object directly to the page. Keep in mind the navigate method provides great flexibility for passing data from one page to another. For more information on using navigation in Windows Store apps, check out the samples on the Windows Dev Center Windows Store apps page (bit.ly/153fDXM).

Before you run your app, add the styles shown in Figure 3 to home.css (/pages/home/home.css) to give the app a slightly cleaner look.

Figure 3 Styles for Home.css

.homepage section[role=main] {
  margin-left: 120px;
  margin-right: 120px;
}
.homepage item {
  height: 225px;
  display: block;
  margin-bottom: 5px;
  margin-top: 5px;
  margin-left: 5px;
}
#hundredYearsListView {
  height: 90%;
}

Run your project at this point, and you should see something that looks like Figure 4.

Chronicling America Home Screen
Figure 4 Chronicling America Home Screen

I’ve built a home screen and a bit of plumbing. The next step is to view the pages of a publication and create a details screen.

Viewing Specific Pages

Adding a details screen is simple with the existing app setup. First, I need to add a new function to data.js to look up the pages of an issue. Next, I need to build out the page control that will actually display this data.

Start by going back to data.js and replacing the comment, “Get function goes here,” with the code in Figure 5.

Figure 5 Get Function for Data.js

function get(url) {
  pagesList.length = 0;
  var uri = new Windows.Foundation.Uri(
    url.replace(/\/seq\-[1-999]{1,3}(\/|\.json)?$/g, ".json"));
  httpClient.getStringAsync(uri)
    .then(
    function complete(result) {
      var editionItem = JSON.parse(result);
      editionItem.pages.forEach(function (item) {
        item.thumbnail_url = item.url.replace(".json", "/thumbnail.jpg");
        item.medium_url = item.url.replace(".json", "/medium.jpg");
        pagesList.push(item);
      })
    })
}

This code takes a URL and uses standard URL conventions of Chronicling America to replace the reference to page one (/seq-1/) with a JSON call to the edition (converting ed-1/seq-1/ to ed-1.json). To do this, use a regular expression that matches the pattern /seq-[1-3 digit number]/ or seq-[1-3 digit number].json and replace it with just .json. By calling the edition’s JSON object, I get information about the issue and not just a specific page. I could go further up the URL to get more general information about the issues or publication in general, but for now I’ll stick to the edition and pages. After adding the Get function, be sure to register it in the WinJS.Namespace.define function as Get so that other pages can use it for reference.

Creating a Details Page First, under the pages directory, create a folder named “details.” This folder will hold the three files that make up a Page control (the HTML, JS and CSS files). Add a new item to the folder, select Page Control (under Windows Store), and name the item details.html. Now update the body tag of details.html (/pages/details/details.html) by using the code in Figure 6.

Figure 6 Code for Details.html

<div class="listview-itemtemplate"
  data-win-control="WinJS.Binding.Template">
  <div class="item">
    <img class="item-image" src="#" data-win-bind="src: medium_url" />
  </div>
</div>
<div class="details fragment">
  <header aria-label="Header content" role="banner">
    <button data-win-control="WinJS.UI.BackButton"></button>
    <h1 class="titlearea win-type-ellipsis">
      <span class="pagetitle"></span>
    </h1>
  </header>
  <section aria-label="Main content" role="main">
    <div id="flipview" data-win-control="WinJS.UI.FlipView"></div>
  </section>
</div>

Notice this code is quite similar to home.html, except I use a FlipView control instead of a ListView control. This gives the app more of the feeling of an eReader. Although I’m discussing only images, Chronicling America provides access to PDF content, Optical Character Recognition (OCR) text and other types of content for consumption. Think about how these different types could add more depth to an app (such as assistive technologies that rely on OCR capabilities) for a future project.

The HTML is complete. Now head over to the CSS file (/pages/details/details.css) to add the cleanup styles that will make the page look decent:

.details section[role=main] {
  margin-left: 120px;
  margin-right: 120px;
}
#pageListView {
  height: 90%;
  width: 90%;
}
#flipview {
  height: 90%;
  width: 90%;
}

Connecting Details to the FlipView

Open the details.js page (/pages/details/details.js) and add the following code to the ready function of the page control:

// Step 1: consume the options data
var selectedItem = options.selectedItem;
// Step 2: acquire data for pages
Data.Get(selectedItem.url);
// Step 3: set the page title
var title = document.querySelector(".details .pagetitle");
title.textContent = selectedItem.label;
// Step 4: set up the ListView to display the pages
var fV = document.getElementById("flipview").winControl;
fV.itemDataSource = Data.PageList.dataSource;
fV.itemTemplate = document.querySelector(".flipview-itemtemplate");
WinJS.Binding.processAll();

Much of this code is the same as in home.js, with the exception that I use options.selectedItem. This object comes from the homepage and is passed into the details page through the navigate function. I use data from the options.selectedItem object to call Data.Get and populate the pageList object for binding to the FlipView control. Just like the ListView, the FlipView uses a WinJS.Binding.Template control to display content. I decided to use a FlipView because I want to display only one page at a time, and that page is a picture. Using the controls for a FlipView also provides a more natural reading experience to users.

Run the code, and you should see something like Figure 7.

Sample Details Page
Figure 7 Sample Details Page

With the details in place, there’s one more piece to wrap up for the app: search capability.

Searching Content

The ability to search newspaper content provides a great research tool. It’s a definite improvement from the microfiche devices of old. By using OpenSearch, Chronicling America provides a powerful search capability that follows an open standard. For my app, I’m going to search against page content using the parameters from 1.usa.gov/188T1HB.

Begin by adding a folder in the pages directory named “search.” Add a new item to the search folder and select Search Results control. To enable search, you must add the Search declaration to your appmanifest and a reference to the searchResults.js file to the default.html file. There’s more to search than adding a few files, so if you aren’t familiar with the Search contract process, you can get started by reading the Windows Store apps Dev Center article at bit.ly/HMtNCo.

The first element needed to build the search capability is to add a new Search function to the data.js (/js/data.js) file. Open data.js and add the code in Figure 8 after the get function.

Figure 8 Search Function

function search(terms) {
  itemList.length = 0; // Clear the list items
  var uri = new Windows.Foundation.Uri(
    urlPrefix + "/search/pages/results/?andtext=" + terms + "&format=json");
  return httpClient.getStringAsync(uri)
    .then(
    function complete(result) {
      var searchList = JSON.parse(result);
      imageList.pop();
      searchList.items.forEach(function (item) {
        item.medium_url = urlPrefix + item.id + "medium.jpg";
        item.thumbnail_url = urlPrefix + item.id + "thumbnail.jpg";
        item.label = item.title;
        imageList.push(item);
      })
  })
}

This simple function takes search terms and builds out the URI to work with the page search URL. The andtext parameter lets you provide search terms to use for content on the page, but remember to check the search definition XML file for more parameters that could be used. To complete data.js, add Search as an item in the namespace so that searchResults.js (/pages/search/searchResults.js) can access it.

Modifying the Search Results Control When you add the item, three files are created (just as with a page control): an HTML file for displaying your results, a CSS file for styling your results and a JavaScript file for binding your results. Update the searchResults.html file first by replacing the itemtemplate to match the data model, as shown in Figure 9.

Figure 9 Updating SearchResults.html

<div class="itemtemplate" data-win-control="WinJS.Binding.Template">
  <div class="item">
    <img class="item-image" src="#" data-win-bind=
      "src: backgroundImage; alt: title" />
    <div class="item-content">
      <h3 class="item-title win-type-x-small win-type-ellipsis"
        data-win-bind="innerHTML: title searchResults.markText"></h3>
      <h4 class="item-subtitle win-type-x-small win-type-ellipsis"
        data-win-bind="innerHTML: publisher searchResults.markText"></h4>
      <h4 class="item-description win-type-x-small win-type-ellipsis"
        data-win-bind="innerHTML:
        place_of_publication searchResults.markText"></h4>
    </div>
  </div>
</div>

Now open the searchResults.js file. This file has a number of functions in it, but most of the changes you make are to the _searchData function. Replace the code for _searchData with the code shown in Figure 10.

Figure 10 Code for _searchData

_searchData: function (queryText) {
  var results;
  if (window.Data) {
    Data.Search(queryText).done(
      function complete(result) {
        if(Data.ImageList.length > 0)
          document.querySelector('.resultsmessage').style.display = "none";
      });
    results = Data.ImageList.createFiltered(function (item) {
      return ((item.title !== undefined && item.title.indexOf(queryText) >= 0)
        || (item.publisher !== undefined && item.publisher.indexOf(queryText)
        >= 0) || (item.place_of_publication !== undefined &&
        item.place_of_publication.indexOf(queryText) >= 0));
    });
  } else {
    results = new WinJS.Binding.List();
  }
  return results;
}

If you compare this code to the code that was generated, you’ll see all I did was update the item object’s data model to reflect the components and add some functionality to occur after the Data.Search function completes. Using the pre-generated code, you can get an app like this to market more quickly. You can focus on creating the specific features of your apps and not have to separately hook up each piece every time.

Before you run the code, update the filter _generateFilters function in searchResults.js to reflect the data model. By default, two filters are added for Groups. Update those to be something more useful for the data model such as a filter on publication city values. With this work complete, you should be able to run your code and see something similar to Figure 11.

The Search Results Screen for the Chronicling America App
Figure 11 The Search Results Screen for the Chronicling America App

To wrap up the search screen, you need to view the details of an article to see its pages. Update the _itemInvoked function in search­Results.js (/pages/search/searchResults.js) with the following code:

_itemInvoked: function (args) {
  args.detail.itemPromise.done(function itemInvoked(item) {
  var searchResult = originalResults.getAt(args.detail.itemIndex);
  WinJS.Navigation.navigate(    "/pages/details/details.html",
    { selectedItem: searchResult });
  });
},

This code captures the currently selected search result item and passes it to the details page. The WinJS.Navigation.navigate function can take two arguments: the address of the page and an initial state declaration. By using the initialState object, you can pass parameters from one screen to the next. In this case, the navigate function passes the selected searchResult object to the details page just like I used the selectedItem from the homepage.

Microfiche for the 21st Century

Chronicling America archives historical newspapers and makes them available through a simple API. In this article, I explored the core capabilities of this Web service and how to consume it in a Windows Store app by using some basic controls. Windows 8.1 provides many opportunities for you to be creative and rethink how people interact with technology.

In this case, I took what would’ve been a sheet of newspaper images and made it into an interactive modern app. You could extend this example to share and store the user’s personalized search settings based on his interests—and this is just the tip of the iceberg. The Library of Congress and National Endowment for the Humanities have provided a rich API on which to build. You can add this historical data to your cookbook for building great apps.

Debrief: Engaging Experience, Reusable Skills

Windows Store apps let you unleash your creativity with data. In this app, you can explore historical newspaper information that could be used for research, education or other historical inquiries. Through a media-rich API and simple JavaScript coding, combining Chronicling America with Windows 8.1 provides an easy way to build a deeply engaging experience.

IT Brief:

In this article, data is pulled from an API and searched, and the content is explored—all by reusing a core set of skills. An engaging experience is created through code reuse and a media-rich API.

  • Engage your development team members by using their existing skills to build rich experiences for users.
  • Engage your users by connecting to Windows 8.1 charm features such as Search.
  • Provide a new vision of an old technology as a market differentiator.

Dev Brief:

Development is a creative exercise, and Windows Store apps give you a lot of freedom. In this article, a retro technology is reimagined and a core set of skills (ListView and Windows.Web.Http.HttpClient) is reused to build it out. Integrating the app with search features allows users to explore the app’s content from anywhere in Windows 8.1.

  • Chaining promises to build asynchronous code workflows.
  • Reusing ListView skills for FlipView controls.
  • Using the JavaScript prototype capabilities to dynamically extend objects.

More Information:


Tim Kulp leads the development team at UnitedHealthcare International in Baltimore. You can find Kulp on his blog at seccode.blogspot.com or on Twitter at twitter.com/seccode, where he talks code, security and the Baltimore foodie scene.

Thanks to the following technical expert for reviewing this article: Eric Schmidt (Microsoft)
Eric Schmidt is a Content Developer in Microsoft’s Windows Developer Content team, writing about the Windows Library for JavaScript (WinJS). When previously in the Microsoft Office Division, he built code samples for the apps for Office platform. Otherwise, he spends time with his family, plays string bass, builds HTML5 video games, or blogs about plastic building toys (https://historybricks.com/).