For every Twitter enthusiast, there was, I suspect, a point of realization that this thing could actually be incredibly useful. Some have cited the plane-in-the-Hudson story as their aha! moment. For me, it was less of a moment and more of a gradual understanding. I began to see its potential as a real-time information source when I first learned of a few important news items — both big international stories and news of a more personal nature — through Twitter.

I began following like-minded people for the interesting links they would post. Before long, information overload took hold. I tried to cull my follow list so I could read everything. I worried I would miss something. Finally, I learned to embrace the firehose and not try to process the whole stream.

But still I thought there must be a better way to separate signal from noise. And then I noticed that the most interesting and important items were appearing maybe three or four times in my Twitter feed. Since then, I’ve wished for a way to mine my feed for those links.

Last week I heard about TwitterTim.es and was thrilled to find it does exactly what I wanted. I spoke with Maxim Grinev, the project’s technical lead, about TwitterTim.es and where it’s headed.

How does TwitterTim.es work?

We look at the tweets that your friends send, and also tweets that friends of your friends send. So, first circle and second circle. And then we extract links from those tweets. Usually links are shortened, so we get the long versions. Then we group by links and calculate how many times each link is posted by your friends and friends of friends to built your personalized “newspaper”. (NB: Links posted by friends get more weight than links posted by friends of friends.) Right now, every “newspaper” is updated about every half an hour. It can be updated more frequently, but we don’t want to stress Twitter.

How did the project start?

As usual, it was a side project. We had been working on some semantic search technology. It’s about using semantic relationships extracted from Wikipedia to organize other data (blogs, news, etc.). As we were working on this, we started using Twitter. We didn’t have any idea in advance of what we wanted to build. We just analyzed how people used Twitter, what information could be detected. We understood that Twitter is not only good for spreading news, but it’s also a good voting system. So we can collect and analyze how many times links are voted on in Twitter. Analyzing this data, we can understand how important this link or this event is.

Who is working on the project, and what’s your business model?

We have 5 people working on this project: 4 developers located in Moscow, and one business guy in San Francisco. We are computer scientists, and we specialize in data management. We are self-funded; there are no external investors. As concerns the business model, we are considering various partnership schemes and selling advertising on TwitterTim.es but we have not decided on anything yet. Now we are mainly focusing on attracting users.

Will TwitterTim.es take advantage of Twitter’s new lists feature?

Right now we don’t do anything with lists. We are thinking about how to incorporate this. One of the options could be to generate newspapers based on some list. So if you have a list of people, you can collect the second-circle friend-of-friend information and build a newspaper for a list.

What other things are in the offing for TwitterTim.es?

We are currently collecting feedback from users. Usually our users request relatively small features — for example, they want to improve the retweet feature. We are going to handle this feedback and add features. In addition to that, we are planning to extend the system in two ways: First, we want to extend the sources that are processed — so, in addition to Twitter, we are thinking about collecting posts and links from Facebook, mainly, and maybe Friendfeed. Second, we are going to allow ranking of news by global popularity. So you would have two different tabs: The first tab is personal news. The second tab is global news. In this sense we will compete with Tweetmeme.

What are your thoughts on the future of news?

I can’t say how it will be. I can just share my own experience, and I think it’s typical: Since I started using Twitter, I’ve nearly stopped collecting news from other sources. Before, for example, I watched news on TV and read more magazines. Now I get nearly all my news from Twitter. I’m quite confident that if I read Twitter, I will not miss some important piece of news. So if a war has started, or there’s some disaster, it will be mentioned at least once in my Twitter timeline.

I have heard a lot of discussion about media sources dying — The New York Times has problems, etcetera. Of course, I think that all these major newspapers and magazines are very important, because journalists have the ability to travel places and work at this full-time. But with regard to selecting what I will read, I’m not going to visit The New York Times website, for example. If there’s some interesting and important article posted there, I will find it in my Twitter timeline.

Also, by the way, there’s an interesting idea we’re looking at: When you visit The New York Times website, for example, you might be interested in getting all the links published there, but ranked according to the judgment of your friends and friends of friends. So it’s the same as TwitterTim.es, but restricted to a single source — The New York Times, in this case. We are talking to one major newspaper about this.

Most commentary has focused on the question of why his post was removed from the Globe and Mail’s books blog, In Other Words. I’ll let others tackle that angle. What I’m interested in is whether the writer, Peter Scowen, has a point. I believe he does, even if it’s poorly expressed:

Last week, our headline on the review for Nicholson Baker’s The Anthologist was one of those sweetly goofy and slightly shopworn plays on words that newspapers are rightly famous and infamous for. The book is about a self-doubting poet in midlife crisis mulling (and procrastinating) over an essay about rhyme; the headline was “The marinating of the ancient rhymer.”

…

Our merriment came to a screeching halt on Tuesday after I went to a seminar on search engine optimization and discovered that it was actually a really really crappy headline. I learned that this kind of badinage, so peculiar to newspapers, has no place on the Internet. The reason is both simple and deranged: The most important reader of Internet news headlines is not you, the sentient, curious human being, but the robots at Google that scan headlines and return search results based on what their cold, lifeless eyes tell them.

…

Above all, we were taught that Internet headlines have to be written with a certain kind of hipster doofus in mind. This person was embodied by the groovy, ever-pacing journalism professor who led the class on writing for robots (he didn’t call it that), and who whipped out his iPhone and boasted that he will not click on anything whose headline doesn’t hand the story to him on a digital platter.

I happen to know the journalism professor in question. His name is Alfred Hermida, and he is anything but a “hipster doofus”. He’s a keen observer of the changes taking place in the practice of journalism, and I’m happy to be joining him on the faculty of the University of British Columbia in January.

I have taught on the subject of headlines and findability, both at the L.A. Times and for Poynter’s NewsU, and I have always stressed this point: It’s not about writing for Google. It’s about writing for humans, with search engines in mind — a theme Alf says he raised in his seminar. But if we’re going to write with an eye toward findability, we have to understand how search engines work and how people use them, and I presume that’s why The Globe and Mail invited Alf to speak.

I wasn’t there, but I suspect there may have been some nuance in Alf’s presentation that was lost on Scowen. In any case, there are ways — both technical and editorial — to publish great headlines without killing search relevance:

• At the very least, most content management systems these days will allow editors to write a literal, search-friendly headline for the story and put a more creative, punny headline on their homepage and section fronts, where keywords don’t matter as much.
• Better yet, if your CMS supports it, you could put your literal headline in a story’s <title> tag and on RSS feeds, and get more abstract in the display headline that readers see when they pull up the story. The New York Times has been doing this a bit, I’ve noticed.
• Finally, if you have control over the words in a post’s URL — and with many blogging tools you do — you can put full names and keywords there instead of in the headline and still get them seen by search engines. Mashable, a popular blog on social media, seems to be optimizing the URLs on its posts in that way.

Scowen raises an important topic, but he cheapens his argument by suggesting that readers who arrive at news content via search — about a third of the audience of many news sites — are intellectually incurious and that journalists who cater to them are dumbing down the craft.

(For what it’s worth, I think his post does kind of fit in a blog about books, because it captures an important difference between online and print writing. And readers’ reactions could have been illuminating for the Globe and Mail staff. It’s a shame that this conversation wasn’t allowed to take place on the G&M site and had to happen elsewhere instead.)

But what else is there? The folks at BusinessWeek think they have an answer, and it’s not about how much content users consume but rather what they do with it. I asked BusinessWeek’s online editor, John Byrne, about his team’s efforts to go beyond pageviews and visits to quantify something more inscrutable: user engagement.

What is BusinessWeek’s definition of user engagement and why is it important?

User engagement is how we nurture and build a community. Our reader engagement index is a comments-to-postings measure for a given month: So we will tally how many comments on X number of stories/blog posts that BusinessWeek.com published that month. This gives us a ratio figure that we track to determine our monthly reader engagement index and growth. In February of this year, we received from our community 28.2 perspectives and insights for every story or blog post we published. A year earlier, we received 23.7. So we know we’re moving in the right direction.

It’s important because we value, and so measure and gauge, all our interactions with our readers on BusinessWeek.com — including commenting on a story or blog post. The next level is how our writers and editors engage our readers in a conversation, and also welcoming our readers to write longer pieces for us, or to report (at least once a week) a reader-suggested story. We’re also engaging with BW readers on other sites, such our Ning network that served as a forum to generate and debate stimulus spending priorities for the Obama administration, or interactions involving our 50+ staffers on Twitter. If we don’t listen to our readers and interact with them, and then act on the feedback and suggestions they’re giving us, we’re dead in the water. That applies to any media brand today, not just BusinessWeek. We’re just making it more of a priority, including featuring readers on an equal plane with our writers — on our home page, for example, our featured reader is given more prominence than even a Jack & Suzy Welch.

What information sources and tools do you use in measuring engagement?

Our reader engagement index involves Omniture (for stories) and Movable Type to track numbers of blog comments and posts.

Beyond our reader engagement index, other measures include how much you’re retweeted — for instance, one of my tweets on March 25 was retweeted 130 times. We also look at referring traffic from blogs or Twitter on Omniture, or by running a Twitter search: http://search.twitter.com/search?q=businessweek or http://search.twitter.com/search?q=johnabyrne.

And also look at Google BlogSearch or blogpulse (owned by Nielsen Buzzmetrics) for mentions of businessweek.com: http://www.blogpulse.com/search?query=www.businessweek.com

What do engagement metrics tell you that conventional metrics do not?

It shows, in quantifiable/measurable terms, how much our readers care about us. To post a comment or submit a suggestion is a strong indicator of a BW loyalist, someone we need to nurture and engage and reward. It also tells us how much (and how well) our staffers are interacting with readers. The problem with time spent on a site is that it also measures, in the case of a portal, email time, or in the case of a site heavy on video, time spent watching video, which can be like TV. I also argue that simple pageview metrics are heavily influenced by slideshows and email. There is no better sign of commitment or engagement than the act of reading a substantive piece of journalism, thinking about it and then forming a point of view on that story that you’re willing to write and share with others. That is true engagement.

How do you use this information to improve the site?

You can’t manage something if you don’t measure it. So having a point of reference for exactly how we’re doing drives other ideas and initiatives to increase engagement.

Does increased user engagement translate into benefits for advertisers?

Yes. Anecdotally, our sales team is selling our engagement story and using it to differentiate what we do versus our competition. It also helps to better position our Business Exchange, a new Web 2.0 product we launched last September, as a key component of our engagement efforts.

So, let’s look at some potential engagement metrics and what they might tell us. This is by no means a comprehensive list; it’s just what came to my mind. If you have other thoughts on ways to measure engagement or how you might use this data, please, um, “engage” in the comments below.

• Internal metrics: Statistics about engagement that takes place on your site
• Comments posted: Shows how much users are inclined to react to a topic, or supply insights of their own.
• Return commenters: In other words, how many people comment multiple times on the same item? This is a measure of conversation around a topic. (Kudos to the Guardian’s Kevin Anderson for this idea.)
• Times e-mailed: Reveals how often users are sharing this information with friends. This metric probably skews toward neophyte users, as more experienced users are presumably less likely to use an “e-mail this” feature.
• Average time spent on page: Shows how thoroughly users are consuming the content, perhaps? Lots of asterisks, though, as John points out.
• External metrics: Statistics about how people share and discuss your content elsewhere
• Tweets/retweets: Measures how “viral” this content is in a social network. There’s also geographic information embedded in these tweets that could tell you where a topic resonates particularly strongly.
• Diggs: Another measure of the viral nature of a topic. Given Digg’s audience, this metric might favor content that appeals to a techie crowd.
• Delicious saves: Shows how many users stored this page with an eye toward returning to it. This metric could be particularly useful for ongoing features that you want to build a regular user base for.
• Inbound links from blogs: Quantifies the discussion taking place in the blogosphere. This could help you identify the blogs that are most attuned to the content you produce — as opposed to just the ones that send you the most visitors (which are not necessarily the most engaged users).

Each of these metrics is easily available for a given URL at a given moment, but keeping track of all your stories over time would be impossible without some automated assistance — particularly with regard to the external metrics.

Here’s what I’d like to see: A web service that will track a URL across several services (Technorati, Delicious, Digg, and maybe internal analytics packages too) to see how it’s being referenced in each medium, then tabulate all those metrics into a single “engagement score”. (And I’d love to hear from any programmers who want to take a stab at building this!)

Meanwhile, anybody have any engagement metrics tips they’d like to share?

Rather than take a hipshot off those headlines, though, we're going to be proactive on OJR this week, starting with this piece from Eric Ulken, who offers a roadmap for established news organizations to enliven their online efforts.]

In a nondescript training room in the BBC’s White City building in West London, about 80 people are huddled around tables with placards bearing names like “Dr. Who” and “Top Gear” [BBC TV show titles], engaged in discussions on topics ranging from user-generated content to alternate-reality gaming.

The assembled thinkers and tinkerers represent many different arms of the British media behemoth, from radio news to Web production to technology. About the only things they have in common besides an employer are an interest in innovation and an eye to the future.

They’re taking part in the second BeeBCamp, an “unconference” in the tradition of BarCamp (and partly inspired by the Guardian’s GameCamp) that aims to bring together forward-thinking staffers and a few outsiders to talk about themes loosely related to the future of the BBC. [Disclosure: I was one of those outsiders, and, in the everybody-pitches-in spirit of the unconference, I talked about my work in data journalism at the L.A. Times.]

BeeBCamp, according to the BBC blog’s write-up of the event, “is designed as a collective, spontaneous bashing together of ideas, with no set structure to the day.” A whiteboard goes up first thing in the morning, and anybody who has an idea for a discussion or presentation claims a spot on the schedule. For example, one participant wrote: “We own twitter.com/bbc. What should we do with it?” (Some ideas here.)

On the whiteboard with the morning schedule, each show title corresponds to a table. Here’s a shot of the afternoon schedule.

I’m a little late with this post, as it’s been almost a month since the Feb. 18 gathering. There’s already ample coverage of the discussions and presentations (plus tags on Twitter and Flickr), so I won’t rehash all that. Instead I’d like to consider the broader idea of BeeBCamp and similar gatherings as they relate to the need to foster innovation in traditional media organizations. BeeBCamp and events like it are great examples of how “big media” — often seen as bureaucratic and impenetrable — can break down walls, open themselves up and facilitate the development of new ideas.

Why might a media company want to host an event like this? Some reasons:

• Silo-busting: BeeBCamp brings together staffers from disparate parts of a huge institution — folks who might never have a business reason to talk to one another but whose goals and interests mesh, often in unexpected ways. (I got the feeling a number of the BeeBCamp participants had never met before.) The interdisciplinary nature of the gathering is what makes it so useful, as experts apply their unique perspectives and skills to common problems.
• Openness: Everything at BeeBCamp is on the record, unless somebody holds up a sign that says “unbloggable”. This means a lot of what is said will get rebroadcast and commented on by people outside the organization, which is, at the least, a way of showing the world that the BBC is thinking and talking about the future, and at best a way to engage in an informal dialogue with the audience.
• Innovation: Sometimes it’s useful to get away from the desk for a while and talk informally with colleagues. Not the ones you sit next to, but the folks across the building (or across town, or across the country) whom you wouldn’t ordinarily interact with. Crazy, silly ideas flow, which beget less silly ideas, which occasionally lead to completely sane and doable ideas. And because people are free to blog the discussions, there’s a good record of what’s said, which can be a useful starting point for follow-up discussion and action.

BeeBCamp is just one example of how media organizations are opening up the process of innovation. Here are some formats that have been used:

• Hack day: This concept, which originated at Yahoo, typically calls for giving techies (often working in concert with product and content folks) 24 hours to build an idea into a functional prototype. After trying out the format internally in 2005, Yahoo conducted the first open hack day in 2006 and continues to do both internal and public hack days. Matt McAlister, one of the instigators of hack day at Yahoo, is now at the Guardian, which did its own internal hack day (with a few outside guests) last year. McAlister has a round-up of the results, complete with video highlight reel, on his blog. (I’d be interested in hearing if other media have hosted hack days.)
• Meetup: The Chicago Tribune has been making good use of meetups (or tweetups, i.e., meetups organized via Twitter) to engage in informal dialogue with readers. It works like this: The Trib (in the persona of Colonel Tribune) invites local bloggers, twitterers and interested readers of all stripes to meet – no agenda — usually at a local bar. The result: Ideas direct from readers, kudos in the blogosphere and good karma all around. Last year the Trib also invited local bloggers to tour the paper.
• Unconference: BeeBCamp, BarCamp and the recent regional NewsInnovation BarCamps fall into this category. Here’s how you might organize an unconference in your organization: Find interested colleagues. Bring in some clever outsiders. Get them talking about the future and see what happens. Make it clear to people that what’s said is on the record. You want folks to feel free to blog and comment about what they see and hear, for reasons mentioned above.

So, what’s the result of all this interaction? I asked Philip Trippenbach, the BeeBCamp organizer and the BBC’s “serious games” guru, if he had examples of products that have come out of BeeBCamp discussions. His response:

Sorry, but I can’t give you any examples of where this has happened at the BBC — yet.

There is one good concrete idea that came out of BeeBCamp: setting up a BBC-wide innovation database. Prototyping this is trivial, and that’s happening, but the tough thing is going to be overcoming the institutional/bureacuratic hurdles to implementation.

However, this isn’t to say that BeeBCamp has had no impact — far from it. I can’t count the number of interdepartmental contacts and discussions that arose as a result of it. This is the sort of interaction that leads to better cooperation and information-sharing across the company. It’s not to be underestimated. Many, many projects coming out of different departments will be informed by this kind of information-sharing through the company. What’s more, I know of two other events that are being planned in the wake of BeeBCamp, and with the same aim: get more New Media people from across the corp sharing and getting to know each other.

What an event like BeeBCamp and its successors does is wake up the community. It takes a community to raise a child, it takes a community to find a phone, and it takes a vibrant, active, connected community to break new ground in media. That’s what we’re doing with BeeBCamp: stoking the flames, so the embers can forge steel.

If you’ve held events like these to promote innovation in your organization, please share your experience here. And if your company hasn’t started bashing together ideas this way, why not be the catalyst? If the BBC can do it, so can you.

I’m no economist, but I think the problem comes down to this: The Internet is a single, efficient market governed by the laws of supply and demand*. Because there’s surplus ad inventory online — particularly low-grade inventory — prices are falling. But what if the surplus inventory is largely the result of a glut of duplicative content? Would the problem go away if news organizations simply stopped doing about half of what they do and focused on the stuff nobody else is producing?

Consider a scenario: Newspaper A posts a local scoop to its website. The story is picked up by other news organizations. It’s rewritten, repackaged, sent out on wires, and within hours that story or some version of it — sans additional reporting — is on a hundred different websites. Much of this duplication is automatic, but some of it is done by human editors. (See Google News any day for an example of this.) Best-case scenario, a few of those sites actually link back to Newspaper A.

Now let’s say most of the duplication stops. Because there are fewer versions of the story, more eyeballs now find their way to the original scoop on Newspaper A’s site. Good. But aren’t many of these additional eyeballs just single-page, out-of-market visits that have little value to advertisers? Maybe, but if Newspaper A is sticking to its core mission of covering local news, it will be able to deliver an audience that’s more cohesive on the whole — and therefore more sellable — than if its content is all over the map.

Those of us who have worked for years in online news remember a time when repackaging news from all over was a large part of what we did. At some point most of us figured out it was a waste of time. But sadly, there’s still a lot of duplication going on in mainstream media websites, in part because it’s seen as necessary for a newspaper to be a broad and semi-comprehensive sampling of the day’s news and information.

Well, no more. You want comprehensive? Go to the BBC.

If newspaper bosses are serious about preserving the kind of journalism that makes newspapers great, here is what they must do right away:

1. Stop wasting time on stuff other people are already doing. This means focus obsessively on local or topical content. The era of the newspaper as bundler of many varieties of content is over. If you cover a community, do nothing that doesn’t relate to that community. If you cover a topic, do nothing that doesn’t relate to that topic.
2. Stop syndicating valuable content to other websites. Let them link to you. (And for goodness’ sake, link out. Do it for the karmic rightness of it all, or do it because it adds significant value to your own content. However you justify it, putting your stuff squarely into the clickstream is essential to staying relevant. You can’t just be the endpoint.)
3. Scale back or cancel wire service agreements. They’re not helping your online product and they might be stealing value from your own content. I have a lot of respect for The Associated Press and the work that all wire-service journalists do, but I just don’t think the AP’s ownership structure and funding model make sense anymore. (If Reuters can thrive as a standalone news organization, maybe AP can too. But newspapers can no longer afford to subsidize the creation of content that doesn’t benefit them directly.)

Am I saying I think newspapers can increase the value of their content to advertisers simply by reducing inventory, the way OPEC does for oil? No (and we can see how well that strategy’s worked for OPEC recently, too). Ad inventory, unlike oil, is not a fungible commodity. This isn’t about reducing inventory in general. It’s about reducing low-value inventory: all those impressions from random walk-ins who aren’t a sellable audience because they have nothing in common.

We talk about the newspaper’s unique status as a profit-driven public trust and the threat that ongoing structural changes pose to that fragile duality. But how big does a newspaper actually need to be in order to fill the public service role we ascribe to it? Could the Los Angeles Times effectively and profitably cover Los Angeles with, say, 300 journalists (half its current staffing level)? My guess is it could, if that’s all those 300 people did.

I feel for my dedicated and talented industry colleagues who have lost jobs in the U.S., the U.K. and elsewhere. This disruptive event is clearly a painful one for journalists. But if newspapers make smart choices this year, maybe it won’t be a crisis for journalism.

—

* Most of what I’m saying here has already been said by various people, so it shouldn’t sound particularly radical. After I started writing about supply and demand, I noticed Nicholas Carr’s thoughtful piece along the same lines. And of course, Jeff Jarvis and others have been making the case for linking over syndication for years.

The much-buzzed-about idea aims to develop open standards and APIs to make source documents “easy to find, share, read and collaborate on.” (You can find the full text of the proposal here.)

I asked three of the proposal’s authors, Aron Pilhofer of the Times and Scott Klein and Eric Umansky of ProPublica, to elaborate on their vision for document nirvana.

+++

Can anyone add documents to the repository, or is it necessary to be a news organization? Any concerns over the possibility of forged documents being uploaded?

Aron Pilhofer: The repository will be open for anyone to read from, but not to contribute to. It will be limited to news organizations, bloggers and watchdog groups whose mission includes publishing source documents as a means of better informing the public about issues of the day. That said, the software that makes DocumentCloud go will itself be open source, and available for anyone to use. So, if others want to create DocumentClouds of their own, they can certainly do that.

Scott Klein: We don’t want DocumentCloud to become a generic repository for all documents, or as a quick-and-dirty way to host PDFs. We want somebody to have found these documents to be of news value.

Presumably, DocumentCloud will not be branded with the NYT and ProPublica logos front and center. Would it be staffed and maintained as a separate entity?

AP: There is so much misinformation out there on this question, so I’m glad you asked. In fact, that is what we are asking Knight to fund: the creation of a completely independent entity called DocumentCloud. So the answer, of course, is: It won’t have any NYT or ProPublica branding.

Though we’ve just started to talk about structure and such, it’s entirely possible the only connection the Times, at least, has to DocumentCloud once it’s up and running is as a user and contributor.

SK: Same with ProPublica. Although I suspect somebody from both the Times and ProPublica will be part of the board for DocumentCloud, it’s important to note that this is going to be completely separate from both organizations and shouldn’t monetarily benefit either.

What is the nature of the collaboration between the Times and ProPublica? How will the work on this project be divided?

AP: TBD, but probably I will focus more on the technology side because the Times is contributing a large amount of the software and I understand that part best.

SK: I think we’ll each do a bit of everything but the plan is for the grant to fund developers, so the bulk of the development work won’t need dividing.

The Knight grants come with strings attached (namely, the requirement that projects be open-source) that might turn off some for-profit companies. Aron, how did you sell your bosses on the idea of applying for this? And, as a for-profit company, how would the NYT benefit from this grant?

AP: There’s a bit of misinformation out there about the role of The New York Times in this project, so maybe I should clarify this a bit more.

The grant is not for The New York Times, so the question of strings and for-profits just isn’t relevant. The Times won’t be involved in any way except as a founding participant and donor to the project (contributing my time and a significant chunk of software).

The grant would be used to create an independent, non-profit organization called DocumentCloud, which would manage the grant, build and maintain the software and so forth. Given the intensely competitive nature of the news business, we reckoned that this project had to be in the hands of an independent, impartial broker in order for a consortium like this to work.

DocumentCloud hasn’t been a hard sell because we’re we’re not asking anyone to do anything they aren’t already doing. We (like most media organizations) are already posting source documents online — just not in a way they can be easily searched, cataloged or shared.

If things go well, everyone will benefit because, finally, there will be open standards and open-source technologies available to make that happen. And even if it fails utterly and completely, DocumentCloud will still provide new tools to make publishing documents online easier, faster and more accessible for everyone.

If the proposal is approved, will DocumentCloud be developed in-house, or will you hire outside developers (or both)?

AP: Development will be done entirely by DocumentCloud developers (see above). Part of the grant funding is to support a dedicated development team.

SK: One tidbit that I don’t think we’ve shared widely is that DocumentCloud is designed to live in the cloud (get it?) so we plan to use Amazon’s EC2 and S3 infrasctructure very extensively, and I know Aron’s toying with releasing the DocumentViewer as an EC2 AMI to make it really easy for news orgs to use it without worrying about their content management system or IT people at all.

Seems to me that one of the biggest differences between the DocumentCloud idea and existing document-viewing systems (Docstoc, Scribd, etc.) is the provision to OCR each document, which will allow people to search within documents and to link to and annotate specific passages. Any thoughts on how the OCR part will work?

AP: We outline some of the differences in our latest grant application, but this is really quite a bit more of an apples/oranges comparison than you may realize.

DocumentCloud isn’t a viewer; it’s a standard, and a web service. It’s a system that allows anyone to make documents sharable regardless of what platform it’s on or where it’s hosted.

Scribd is similar in that users can upload documents and make them public. Within Scribd, registered users can comment on those documents, link to them, search them, etc. But everything has to happen within the Scribd environment.

DocumentCloud takes that idea a step further and removes the barriers. It allows users to search, link to and comment on documents regardless of where they are housed, or what platform they are sitting on. All we will ask is that those who are contributing documents do so in a standardized format.

So, Scribd or Docstoc could, in theory, adopt the standard and enable their users to contribute to DocumentCloud, and we hope they do.

I think some of the confusion on this point is of our own making because of the DocumentViewer portion of the project. The viewer is (or will be) nothing more than an off-the-shelf, completely open-source implementation of that standard. But DocumentCloud will be completely agnostic in this regard. If Scribd or Docstoc (or GovernmentDocs.org or The Smoking Gun) want to create their own compatible viewer, they are completely welcome to do so.

The reason we included the viewer in the grant application (and there was a lot of discussion internally about this) is because a key part of this project is lowering the barriers of participation. Many organizations don’t have the capability of developing their own software for viewing documents or integrating them with DocumentCloud, so we felt that was an important part of the project too so we kept it in.

SK: Aron’s making a key point here: This isn’t competitive with Docstoc or Scribd, and isn’t even meant to replace a simple list of PDFs if that’s what you want to use. DocumentCloud is a way to organize all of these disparate ways of storing digitized source documents in a way that makes them maximally useful to “reporters” (counting, of course, traditional newsroom reporters as well as bloggers, academic researchers, etc.) Frankly, DocumentViewer is, for a news organization presenting complex document collections, a really great user experience, but it’s not required to be part of DocumentCloud.

Will DocumentViewer be released to the public even if the DocumentCloud proposal isn’t funded? Is there a timeline for that?

AP: Yes, but there’s no specific timeline right now. We’re working on it in between other, more deadline-specific projects. My best guess right now is that we’ll have something releasable in the late spring. That’s about as specific as I can get right now.

What organizations are you soliciting source documents from? I think Eric mentioned the National Security Archive; anywhere else?

AP: None yet. We have talked to a limited number of groups (Gotham Gazette and, yes, the National Security Archive and possibly others) to partner with us on the development of the project. But we’re not actively soliciting documents at this point.

SK: We’ve got a fairly extensive wish list of news organizations and nonprofit groups we want to bring in on the project (none of whom would surprise you I think), and we’ve talked with some folks very informally but all of our discussions have been like “save the date” cards as opposed to wedding invitations, if you get my meaning.

Eric Umansky: As Aron and Scott have said, we’re just at the beginning of this and have just had initial discussion with a few groups. Having said that, we have been in touch with the NSA (the private, non-profit one) and are particularly excited about working with them since they are really among the best in the biz at cataloging and archiving government source documents.

Are there certain kinds of documents that you think will be particularly well-suited to perusal and annotation using DocumentCloud?

EU: Honestly, I’m not really sure. Like the best parts of the Web, what we’re trying to do is build an infrastructure that will support and encourage intelligent contributions. So, not to get all web doe-eyed about it, but the very utility of it is that people will have the ability and interest to submit documents beyond the one we’re already aware of.

How do you plan to surface the most interesting stuff from within this potentially vast database? Will there be a blog or a recent highlights list of some kind? Will you take some pop-culture cues from The Smoking Gun?

AP: We’re hopeful that users will surface this stuff, and we won’t have to. We have not talked about whether we’ll have a blog or highlights — or even if DocumentCloud itself will have a web presence outside the APIs. It’s just not something we’ve decided yet.

SK: We’re laying the foundation for the great work of others, and have very little interest in applying our own editorial judgment on what people post, assuming two things: 1) people follow whatever rules we come up with (like don’t post inappropriate things, etc.), and 2) they themselves apply editorial judgment to what they upload. I think it’s impossible to predict what kinds of stories this will help tell, and I find that really exciting.

EU: I agree with Scott and Aron. We’re really at too early a stage to have a concrete sense of this. And I’m the farthest one here from the software side of this, but one thing we would like to do is build a kind of reader loop into the system. So, not only could you sort by the “most read” documents but you could also sort specific pages that way. For example, if you had a 500-page report that had juicy bits buried on pg. 432, the “crowd” would eventually point you there since it would be flagged and become the most popular page.

Any updates on the News Challenge judging process? Do you know if you’re in the “top 50″?

AP: No idea.

SK: All we know is that we’ve passed the first of four rounds of scrutiny, as have some other really great ideas.

Winners will be announced in the fall, according to the Knight News Challenge site.

It took several months to secure the tech resources and a couple more months to create wireframes and spec out requirements for what would become the Homicide Map, with the help of a couple of talented developers and a project manager on part-time loan from the website’s IT department. We were fortunate, of course: We actually had access to this kind of expertise, and since then we’ve hired a couple of dedicated editorial developers. I’m aware that others might not have it so good.

Last week, Robert Niles argued that news organizations should be in the business of creating “killer apps”. Put another way, there is a need to develop tools that hew to the content rather than the other way around. But creating the functionality Robert describes takes a closer connection between news thinking and tech thinking than is possible within news organizations’ traditional structures and skill sets.

In this post, I’ll try to squeeze some wisdom out of the lessons we learned in the process of assembling the Times’ Data Desk, a cross-functional team of journalists responsible for collecting, analyzing and presenting data online and in print. (Note: I left the Times earlier this month to work on some independent projects. I am writing this piece with the blessing of my former bosses there.)

Here, then, are 10 pieces of advice for those of you building or looking to build a data team in your newsroom:

1. Find the believers: You’ll likely discover enthusiasts and experts in places you didn’t expect. In our case, teaming up with the Times’ computer-assisted reporting staff, led by Doug Smith, was a no-brainer. Doug was publishing data to the web before the website had anybody devoted to interactive projects. But besides Doug’s group, we found eager partners on the paper’s graphics staff, where, for example, GIS expert Tom Lauder had already been playing with Flash and web-based mapping tools for a while. A number of reporters were collecting data for their stories and wondering what else could be done with it. We also found people on the tech side with a good news sense who intuitively understood what we were trying to do.
2. Get buy-in from above: For small projects, you might be able to collaborate informally with your fellow believers, but for big initiatives, you need the commitment of top editors who control the newsroom departments whose resources you’ll draw on. At the Times, a series of meetings among senior editors to chart a strategic vision for the paper gave us an opportunity to float the data desk idea. This led to plans to devote some reporting resources to gathering data and to move members of the data team into a shared space near the editorial library (see #8).
3. Set some priorities: Your group may come from a variety of departments, but if their priorities are in alignment, disparate reporting structures might not be such a big issue. We engaged in “priority alignment” by inviting stakeholders from all the relevant departments (and their bosses) to a series of meetings with the goal of drafting a data strategy memo and setting some project priorities. (We arrived at these projects democratically by taping a big list on the wall and letting people vote by checkmark; ideas with the most checks made the cut.) Priorities will change, of course, but having some concrete goals to guide you will help.
4. Go off the reservation: No matter how good your IT department is, their priorities are unlikely to be in sync with yours. They’re thinking big-picture product roadmaps with lots of moving pieces. Good luck fitting your database of dog names (oh yes, we did one of those) into their pipeline. Early on, database producer Ben Welsh set up a Django box at projects.latimes.com, where many of the Times’ interactive projects live. There are other great solutions besides Django, including Ruby on Rails (the framework that powers the Times’ articles and topics pages and many of the great data projects produced by The New York Times) and PHP (an inline scripting language so simple even I managed to learn it). Some people (including the L.A. Times, occasionally) are using Caspio to create and host data apps, sans programming. I am not a fan, for reasons Derek Willis sums up much better than I could, but if you have no other options, it’s better than sitting on your hands.
5. Templatize: Don’t build it unless you can reuse it. The goal of all this is to be able to roll out projects rapidly (see #6), so you need templates, code snippets, Flash components, widgets, etc., that you can get at, customize and turn around quickly. Interactive graphics producer Sean Connelley was able to use the same county-level California map umpteen times as the basis for various election visualizations in Flash.
6. Do breaking news: Your priority list may be full of long-term projects like school profiles and test scores, but often it’s the quick-turnaround stuff that has the biggest immediate effect. This is where a close relationship with your newsgathering staff is crucial. At the Times, assistant metro editor Megan Garvey has been overseeing the metro staff’s contributions to data projects for a few months now. When a Metrolink commuter train collided with a freight train on Sept. 12, Megan began mobilizing reporters to collect key information on the victims while Ben adapted an earlier Django project (templatizing in action!) to create a database of fatalities, complete with reader comments. Metro staffers updated the database via Django’s easy-to-use admin interface. (We’ve also used Google Spreadsheets for drama-free collaborative data entry.) … Update 11/29/2008: I was remiss in not pointing out Ben’s earlier post on this topic.
7. Develop new skills: Disclaimer: I know neither Django nor Flash, so I’m kind of a hypocrite here. I’m a lucky hypocrite, though, because I got to work with guys who dream in ActionScript and Python. If you don’t have access to a Sean or a Ben — and I realize few newsrooms have the budget to hire tech gurus right now — then train and nurture your enthusiasts. IRE runs occasional Django boot camps, and there are a number of good online tutorials, including Jeff Croft’s explanation of Django for non-programmers. Here’s a nice primer on data visualization with Flash.
8. Cohabitate (but marriage is optional): This may be less of an issue in smaller newsrooms, but in large organizations, collaboration can suffer when teams are split among several floors (or cities). The constituent parts of the Times’ Data Desk — print and web graphics, the computer-assisted reporting team and the interactive projects team — have only been in the same place for a couple months, but the benefits to innovation and efficiency are already clear. For one thing, being in brainstorming distance of all the people you might want to bounce ideas off of is ideal, especially in breaking news situations. Also, once we had everybody in the same place, our onetime goal of unifying the reporting structure became less important. The interactive folks still report to latimes.com managing editor Daniel Gaines, and the computer-assisted reporting people continue to report to metro editor David Lauter. The graphics folks still report to their respective bosses. Yes, there are the occasional communication breakdowns and mixed messages. But there is broad agreement on the major priorities and regular conversation on needs and goals.
9. Integrate: Don’t let your projects dangle out there with a big ugly search box as their only point of entry. Weave them into the fabric of your site. We were inspired by the efforts of a number of newspapers — in particular the Indianapolis Star and its Gannett siblings — to make data projects a central goal of their newsgathering operations. But we wanted to do more than publish data for data’s sake. We wanted it to have context and depth, and we didn’t want to relegate data projects to a “Data Central“-type page, something Matt Waite (of Politifact fame) memorably dubbed the “data ghetto.” (I would link to Waite’s thoughtful post, but his site unfortunately reports that it “took a dirt nap recently.” Update: It’s back, and here’s the post.) I should note that the Times recently did fashion a data projects index of its own, but only as a secondary way in. The most important routes into data projects are still through related Times content and search engines.
10. Give back: Understand that database and visualization projects demand substantial resources at a time when they’re in very short supply. Not everyone in your newsroom will see the benefit. Make clear the value your work brings to the organization by looking for ways to pipe the best parts (interesting slices of data, say, or novel visualizations) into your print or broadcast product. For example, some of the election visualizations the data team produced were adapted for print use, and another was used on the air by a partner TV station.

When I shared this post with Meredith Artley, latimes.com’s executive editor and my former boss, she pointed to the formation about a year ago of the interactive projects team within the web staff (Ben, Sean and me; Meredith dubbed us the “cool kids,” a name that stuck):

“For me, the big step was creating the cool kids team — actually forming a unit with a mandate to experiment and collaborate with everyone in the building with the sole intention of creating innovative, interactive projects.”

And maybe that should have been my first piece of advice: Before you can build a data team, you need one or more techie-journalists dedicated full-time to executing online the great ideas they’ll dream up.

What else did I miss? If you’ve been through this process (or are going through it, or are about to), I hope you’ll take a minute to share your insights.

Sample electoral vote scenario: (not my prediction; just an uneducated guess for demonstration purposes only)

This is the creation of Sean Connelley, our Flash guru, based on our 2004 electoral vote tracker. The cool addition this time around is the sharing functionality.

We’re hoping to improve on this as the campaign heats up, perhaps adding demographic info and data on past elections by state. Would love to hear suggestions.

Across the nation, more than 4,600 have died while in service to the country. Of the California dead, the median age was 23. Their deaths left 205 widows and three widowers, and 300 children who will grow up without their fathers, two without their mothers. Thirty-eight of the 492 were engaged.

About 67% were in the Army, Army National Guard or Army Reserve; 27% in the Marine Corps or Marine Corps Reserve. The Air Force accounted for 2%, the Navy and Navy Reserve for 4%. Two percent of those killed were women.

At least 59 were immigrants.

I’d like to draw your attention to a new feature that launched on latimes.com this week: The Homicide Map is a visual interface to the Homicide Report, Times reporter Jill Leovy’s effort to chronicle every homicide in Los Angeles County.

As of July 30, The Times has counted 496 homicides in L.A. County. While the Homicide Report focuses on the individual victims, this tool helps users analyze the broader geographic and demographic trends within that staggering figure.

The Homicide Map enables users to: