Thought Leaders in the Cloud: Talking with Aron Pilhofer, Editor of Interactive News Technologies at The New York Times

Aron Pilhofer acts as editor of interactive news technologies
at The New York Times, overseeing a
news-focused team of journalist/developers who build dynamic, data-driven
applications to enhance the Times' reporting online. He joined The Times in
2005. Previously, he was at the Center for Public Integrity in Washington, and
before that at Investigative Reporters and Editors (IRE.org).

In this interview, we discuss:

-The
purpose served by DocumentCloud

-Lack of
technology in newsrooms, and how the cloud is making information more
attainable and process-able by journalists

-How the
elastic capabilities of cloud computing match with the event-based spikes in
demand around news

-How a
"document dump" may cause thousands of documents to appear at one time, and are
better processed by the elasticity of the cloud

-Use of
"CloudCrowd", a Ruby based MapReduce library

Robert Duffner:
Could you take a moment to introduce yourself and to give us some background on
DocumentCloud?

Aron Pilhofer:
Sure. I wear a couple of different hats. At The
New York Times, I'm editor of interactive news, which is a team of
developers in the newsroom who are journalists. What we do is both editorially
and data-driven. We operate like a news desk, but we also were a technology
team.

My other day job is on DocumentCloud, which is a nonprofit
funded by the Knight Foundation. I proposed a grant to fund it with Eric
Umansky and Scott Cline. We were awarded the grant, and we're entering our
second year right now.

The goal of the project
is to improve journalism by creating a site that allows journalists to analyze,
upload, share, and search public source documents that would be otherwise
extremely difficult to find or analyze.

Robert: There's
an old issue in
journalism that journalists often cite documents that aren't available to the
reader. DocumentCloud lets the journalist post those source document in a
public place, so the reader can go back to the source, just as a journalist
could.

As far back as the '20s, though, guys like Walter Lippmann
argued that the public just isn't that interested in the details. Do you find that
people aside from journalists are benefiting from DocumentCloud?

Aron: Actually,
no. Let me just explain a little bit what DocumentCloud is, how it started, and
why the answer is no. There's DocumentCloud the software, which is one part of
what we're building. It sort of sits on top of OpenCalais, which is an open API
that does entity extraction and semantic markup.

Think of it as a set of tools we're providing to journalists
to give them the ability to treat unstructured text more like structured data,
so they can find links between documents that they could not have found through
traditional means.

As an example, think of a case where you send through a
document that includes a reference to the CIA. CIA is meaningful to a human
being. You and I can look at that and go "Oh, that probably means the
Central Intelligence Agency." Or, in other context, it might be the
Culinary Institute of America. It's less clear to a traditional text search.

The Calais engine allows us, in an automated way, to go
through and say "OK, that's the Central Intelligence Agency. And by the
way, here's this other document also about the Central Intelligence Agency, and
both of them reference the same individual that you are curious about."
So, that's an example of some of the tools we're building with journalism in
mind. That's DocumentCloud the software.

Then there's DocumentCloud the community, which is the other
piece of what we're trying to put together. Right now, it includes about 150
journalists and journalism organizations, with that number growing by leaps and
bounds. They're joining the community to use this tool to improve their
reporting.

In order to join that community, you pretty much need to be a
journalist, by our definition. That is, you must be someone whose job, either
paid or unpaid, involves the acquisition, analysis, and ultimately publishing
of public source documents to benefit the public. Normally that means
government documents, and a lot of those documents are acquired through FOIA,
or they might exist on some other site.

Having said all that, we have been approached by any number
of non-journalism organizations, such as law firms. We've gotten the sense that
there is a need out there for sort of a lightweight document management tool,
and we may explore that as a potential revenue generator, but that isn't really
our main focus.

Robert: You
talked about this idea of document management. One of the reasons that self
publishing has been so popular has been the ease by which you can actually
publish to a platform. Can you talk a little bit about how DocumentCloud
removes some of the impediments traditionally associated with IT departments?

Aron: The genesis
of DocumentCloud was from a piece of software we developed at the Times called
DocumentViewer, which is a really straightforward piece of software. It will
take a PDF or a Word document, pretty much anything OpenOffice can open or a
PDF, break it up, extract the text, make it searchable, and then publish it to
the web in an attractive way.

Our thinking going in was
that most news organizations, even the smallish ones, would want something
similar. So our original conception was that DocumentCloud would be sort of the
hub. We would want your metadata, but generally speaking, we thought that all
the member organizations would want this sort of viewer to be on their
hardware, behind their firewalls.

We could not have been more wrong about that, for both good
reasons and unfortunate ones. My perception is that newsrooms lack fundamental
technology to deal with documents, and that is sort of scary. The traditional
way that newsrooms deal with big document dumps is to split them up and have
people sit down with yellow legal pads and pens and highlighters.

That is the highest technology, really, that most newsrooms
currently employ. A lot of newsrooms don't have access to the simplest things,
like OCR. That's surprising to a lot of people, but it's true, and in this
little area of public source documents, we think we can help.

That's why we pivoted early on away from thinking about
DocumentCloud as a federated thing running on hundreds of websites, to a vision
where fundamentally it all goes through us. For the most part, we actually host
the documents on behalf of news organizations.

The way we made that simple and scalable was to make the
entire DocumentViewer portion static. What you see on the website is just HTML,
JavaScript, CSS, and JSon. The only live portion is search, which is provided
dynamically as a service.

All a news organization has to do is get a little embed code
from us, which they can embed anywhere they want in their CMS. They can put it
within a blank page on their own site, in a blog post, or whatever. It's really
simple and really straightforward.

Robert: Some of
these technologies like DocumentCloud coming out are pretty exciting. Can you
talk a little bit about some other ways that the cloud might be fundamentally shaping
journalism practices?

Aron: My team
here at the Times couldn't do what we do without the cloud. We run everything
off of Amazon. On an election night, we can suddenly go from four or five
servers to 22 servers to handle all that traffic. A day later, we can just spin
back down to five servers. There's no way you could do that in a traditional IT
environment.

Robert: One
concern that governments and corporations have about the cloud is where
data is stored. Typically they want or need the data to be stored within
their country's borders. But what's a drawback for some companies in this
scenario actually looks like an advantage for journalism. Is one benefit of the
cloud that it's possible to store any potentially embarrassing government
documents out
of the reach of that government?

[laughter]

Aron: That
thought certainly has occurred to me, and I don't know that it's been
adjudicated anywhere, really.To
flip that idea on its head, consider that in the UK, there's this notion of
Crown copyright, where the public doesn't really own public documents and data.

It's sort of bizarre. For example, postal codes are
copyrighted under Crown copyright, and you have to pay a huge amount of money
to get boundaries of postal codes in the UK. I don't know what would happen if
somebody were to make that data publically available on a server in the US. If
there were some assertion of Crown copyright, would that even apply
jurisdictionally to where that data is hosted?

It's a really good question, and I'm not sure I want to find
out, because this is sort of new territory for everybody. We're pretty cautious
about what we put up on the cloud and what we don't.

Robert: Looking
at DocumentCloud, what was it that required something new to be built? I mean
Microsoft has Office 365
with SkyDrive.
Google obviously offers Google Docs.
There's also Scribd. What did you need
that you didn't find in these existing resources?

Aron: We looked
at all those options early on, and while in 2007, this field obviously wasn't
quite as crowded as it is now, none of them did what we wanted DocumentViewer
to do. DocumentViewer is more than just a way of putting a document online.

For example, it also allows you to do annotations, which is
kind of key from a journalistic standpoint. There's what we have come to refer
to as kind of a journalistic layer on top of a document.

A reporter can go into DocumentViewer, highlight a key paragraph,
click "Drag," and create an annotation. He or she can actually write
a couple of paragraphs to identify the significance of a particular sentence,
phrase, or paragraph and deep link into it.

That allows you to add a narrative to what is effectively a
piece of raw data, and say to the reader, "OK, here's the document that
we're basing our reporting on. But more that, here are the key paragraphs, and
here's why they're key. Here's really what this means."

Scribd didn't do that. Docstoc didn't do that. There was
really no technology we could find that did it in a way that we thought
accomplished our goals. We also wanted something that wasn't Flash-based, which
Scribd at that time was.

Robert: That
makes a lot of sense, particularly to support standards, when you consider all
of the form factors that you can use to access the Web. I imagine various
reporters want to use something like an iPad, a mobile phone, you name it.

Aron: Right. It's
not the world's greatest experience, but you can actually use DocumentViewer on
an iPhone. This is not an anti-Flash rant, or anything like that. It's just we
felt that the right technology for this was to stick to web standards, and what
we've come to refer to as HTML5.

Robert: On your blog, you've talked about how to use
Amazon EC2 behind the scenes. Can you explain how the elasticity of the cloud,
scaling up and down on demand, gets put to use by DocumentCloud?

Aron: Sure. It's
a big challenge. Document processing is a very CPU-intensive process, and so we
needed to be able to scale up rapidly when there's a big uploaded document, so
we did two things. One is that we have built and released a fairly lightweight
parallel processing library we call CloudCrowd. DocumentCloud has actually
released a number of open source libraries. We haven't released the entire
project, but that will come soon.

But the first piece was CloudCrowd, and that was sort of a
lightweight, Ruby-based parallel processing library, which allows us to quickly
add additional processing nodes if we get a 3,000 document dump from AP, which
actually happened last week.

Relatively easily, we can add two, three, four, or 100
servers to the processing pool and split that job up. It's basically a
MapReduce project at that point. So that's how the elasticity helps us on
DocumentCloud. The front end isn't as much of an issue, because once the
documents are actually rendered, it's 100% static content. We just serve those
off of S3.

Robert: Can you
talk about how much you're processing and what you expect that to grow to?

Aron: It
fluctuates, obviously, and it's pretty spiky, which is why we couldn't really
do this in a traditional environment. If you're building a data center, you
have to size it to the biggest spike you expect to have, which means you've got
a lot of time where you're sitting and idling with unused resources. Because we
don't need to worry about that, we can spin up 10, 20, or whatever at a time.

I think the most we've every processed in a day is a few
thousand documents. And then there are certain days where it's just a few
dozen. We opened our beta this summer, and I think we're over 400,000 pages
now, closing in on 500,000.

Robert: You
mentioned already that DocumentCloud uses open source, and is itself open
source.

Aron: Actually,
it's MapReduce, but we don't use Hadoop. Our version of Hadoop is CloudCrowd.
Think of in the old Apple ad, CloudCrowd is Hadoop for the rest of us. It's a
much simpler Ruby-based MapReduce library for doing parallel processing.

Robert: We
definitely sense that investigative journalism is being cut from a lot of news
organizations, because it's expensive and time-consuming. At the same time, computer
assisted reporting, which includes things like web scraping and data
mining, is on the rise and has actually led to Pulitzer Prize winning stories.
Do you think that technology offers new hope to investigative journalism?

Aron: Certainly,
and DocumentCloud, I think, is an example of how technology can be brought to
bear on that. As I said before, most journalists do serious document reporting
and analysis as a very analog process, and I think that the document piece is
just one tiny fragment.

Part of what a lot of computer assisted reporting folks are
doing these days in newsrooms is acquiring the data and making it searchable,
so it's easier for non-technical journalists to work with. I think the smart
application of technology in newsrooms can be a force multiplier for shrinking
staff.

The Times obviously has made a significant commitment to
investigative reporting, which not every news organization has. Anyone who
reads a newspaper knows the industry is struggling, which is a very good reason
why newspaper staffs are shrinking. The way I see it is that technology can
help overcome some inefficiencies, which can help preserve journalistic
quality.

Robert: Hey,
thanks so much for your time. I greatly appreciate it.

Aron: You bet.

Tweet