I love newspapers. I want them to survive, in some form, but it’s important to investigate where the truth in one of the linchpins of the “newspapers need to survive argument” comes from.

Tom Rosenstiel explained this before the Joint Economic Committee hearing on “The Future of Newspapers: The Impact on the Economy and Democracy,” on September 24, 2009:

In every community in America I have studied in 26 years as a press critic, the newspaper in town has more boots on the ground–more reporters and editors–than anyone else–usually than all others combined. A good deal of what is carried on radio, television, cable and wire services comes from newspaper newsrooms. These media then disseminate it to broader audiences.

When we imagine the news ecosystem in the 21st century, the newspaper is still the largest originating, gathering source.

Rosenstiel’s not the only one to make the claim. It’s a common one. John Carroll used to say that 80 percent of news came from newspapers. Len Downie and Robert Kaiser similarly claimed that newspapers were the originators of most content for most broadcast and cable news. And many studies of online blogs show that much of the linking originates from mainstream media, often newspapers.

But are newspapers where it all begins? In an online world, that’s only sort of true.

A study coming out of USC Annenberg of 250 news websites looks at where these sites are bringing information from – whether they are citing the AP or citing their own journalists. Though the analysis isn’t complete yet, initial results seem to suggest that wire services are providing the bulk of news online.

The study, as explained by Annenberg doctoral candidate and researcher Matthew Weber, takes a systems approach. This means that the researchers were taking a look at who was providing information for the network of news organizations, who was doing the filtering for the news organizations, who was collecting the information and from where – and how it was being passed on.

“If you take a systems approach to the news industry, the people who are providing the raw material are predominantly wire services,” he said.

Weber did find that newspapers still are where consumers make their first stop. And while they add their own content, newspapers are also acting as filters – were also bringing in articles from the AP, Reuters, AFP and the like.

“The ‘system’ start with the wires, and ends with the aggregators. Newspapers are jammed in the middle, competing for air,” Weber explained via e-mail.

But when it comes down to who is creating the content for news sites, the organizations providing information were “almost exclusively wire services,” according to Weber.

So wires, in this case, seem to be increasing importance in the news architecture of the online world – and newspapers aren’t the first stop that they used to be, though they do help sort information.

But in some sense, wires have always played an important role that has often been ignored by those who like to say that newspapers have set the news agenda and uncovered the most important stories.

When I was an intern covering cops at the Chicago Tribune in 2003, often my assignment came not from the scanners but from the now-defunct City News Service, a wire service owned by Tribune Co. that sold breaking news to the highest bidder in the local market. The City News Service in Los Angeles, not owned by Tribune, still serves a similar purpose.

Even if we disregard these pre-Internet wires that only operated in a handful of cities, it’s still unfair to say that newspapers set the agenda for the rest of the media in a city. Certainly newspapers often did the rigorous work of providing a detailed account which was then recycled on local news, but television news has never aspired to be anything but a recycling of newspaper headlines even in its golden era.

Cronkite saw his viewers still reading a paper, and today, local news also doesn’t kid itself about being entertainment. The two mediums work more complementary than as leader and follower than we might hope to suggest in our case for news survival.

But there’s a whole other element to where news comes from that has also been ignored in an online context – the world of blogs and online communities – and how this then sets an agenda for newspapers to follow.

Chris W. Anderson, a Nieman blogger and assistant professor at the College of Staten Island – CUNY, has research that suggests that it’s important to look not just at newspapers but at the whole news ecosystem- which includes everything from news to activist communities.

Anderson doesn’t question the macro-level assumption that journalists report and bloggers comment. But he notes that it’s a little more complicated when you look more closely at specific news instances.

Calling them news “blips,” Anderson said, “You’ll have an early period that most journalists wouldn’t call reporting where information will be released in niche spheres of the blogosphere.”

One example he gave was of reports of activists arrests. But it wasn’t that reporters were reading these activist blogs that this news happened to make it into the mainstream newspaper or news media. Instead, journalists got their tips from “being good reporters,” taking cues in the traditional way, perhaps from police or press releases or shoe-leather reporting.

From his observations at the Philadelphia Inquirer and the Philadelphia Daily News, Anderson said, “It’s a misnomer to think journalists are just sitting around reading blogs.”

But once journalists did report on these news blips, these blips were then circulated into the larger blogosphere. But the blips required a certain level of bubbling up to the surface from the niche level of social media, something that happened in traditional ways.

Twitter might make a good case of how newspapers aren’t the first and only source of news, especially on a hyperlocal level. Newspapers may be hoping to compete on the hyperlocal, but this strategy may be questionable especially in cities with actively wired bloggers and tweeters who may have the first claim on news.

My old neighborhood in LA is a Twitter neighborhood. Local stores and restaurants were on Twitter, as are many residents and more active bloggers. We all routinely kept the neighborhood hashtag #DTLA in our posts when commenting about our home. Sure, the bars marketed drink specials to us, but the #DTLA hashtag was the first source of news when the 2009 Lakers celebration got out of hand, then followed by TV and the LA Times. Twitter users provided great on-sight reportage of the Michael Jackson funeral at the Staples Center, often going beyond what mainstream media had to offer.

Did these events wind up back in the newspapers? Sure. But the most active concentration of rumors and new bits of information were coming from a niche community – in this case, the #DTLA one, and in Anderson’s case, the activist community.

Perhaps, instead of staking the claim for newspaper survival on the fact that newspapers provide the first stop of news and set our agenda for what it is we care to talk about, those making the case might start to make a more nuanced argument.

Maybe it’s not as compelling to say that newspapers are the great facilitators of democratic dialogue and discourse instead of the source of all that is news, but it seems to reflect the burgeoning reality of our digital era.

Too many news professionals get bogged down by traditional notions of “journalism,” that what newsrooms publish must be multi-sourced narrative “stories,” from five to 500 inches long, following J-school-approved norms for reporting and narrative structure.

Feh. When I worked a small daily, many of us on staff suspected that the most popular features in the paper were the obits, police blotter and log of ambulance runs. And you know what? When I moved up, to bigger cities and bigger dailies, I missed not being able to check the paper to see where that police cruiser or ambulance I heard yesterday was going.

Readers love information. Whether that’s a police blotter, local bulletin board, school lunch schedule or gripping story in the local paper — they don’t care about the format. Readers just want it to be accurate, relevant and complete. Without anything misleading or extraneous, either.

That’s why I love watching people like Holovaty, whom I’ve interviewed before on OJR. The public has voted with its mouse clicks that it wants more information from the rest of the world that they are finding from the same, stale stories in their shrinking local papers. Holovaty’s creations offer the promise of a reinvigorated news industry, driven by journalists who can wield code, statistics and data every bit as effectively as words and grammar. I e-mailed Holovaty, and asked him about EveryBlock.

OJR: What’s EveryBlock providing that the average Web reader could not get before?

Holovaty: First, fundamentally, we offer a way to browse news at the block level, with a news page for every block — hence the name EveryBlock. We’ve done a fair amount of due diligence and are pretty confident this hasn’t been done before — and in three of the densest cities in America, at that.

Second, we’re providing some information that didn’t previously exist online. Two examples are film locations in Chicago and restaurant inspections in San Francisco. The former is provided to us by the Chicago Film Office, and the latter is provided to us by the San Francisco Department of Public Health, which has its own website but doesn’t include some of the data we publish.

Third, we make it easy to browse information that already existed online but was buried in deep government sites, either in “deep Web” search forms or non-Web-friendly formats such as PDFs. Two examples are landmark building permits in New York City and crime reports in New York City, but there are many other examples across our three city sites. This has been an interest of mine for a number of years, and it’s a dream come true to have the opportunity to do it at this scale.

Fourth, we’re detecting geography in narratives — “blobs,” so to speak — and making it easy for people to find relevant news articles and government documents that refer to specific places near them. Some examples are New York City news articles, San Francisco zoning agenda items and Chicago city press releases. Another (geeky) way to phrase this is that we’re harvesting geographic metadata from unstructured text.

Fifth, we’re providing some light trending and aggregate reports for *each* type of information on our site. For example, see the Chicago crime data.

OJR: Describe the work that went into creating EveryBlock.

Holovaty: The work that has gone into creating EveryBlock has been quite diverse, which makes the job interesting and exciting. On the “human” side of things, we’ve established many relationships with government officials and other partners who are responsible for local data. On the user-interface side, we’ve worked to design a gorgeous, easy to use site and an architecture that accommodates a wide variety of disparate types of information. On the map side, we’ve made our own maps, deciding against Google’s or Yahoo’s map offerings for a number of reasons; that took a sophisticated combination of design, coding and data chops. At the technical level, we’ve developed an array of technology just to get all of this data into an elegant, unified system. It’s beautiful. And we’ve even done a fair amount of manual labor, from hand-drawing neighborhood boundaries to hand-tagging newspaper articles to train our geoparsing algorithms.

OJR: I suspect that when many people, inside and out of the news industry, hear the word “journalism,” they think of a specific, narrative format for providing information. But sites such as EveryBlock provide information outside the traditional newspaper narrative form. Do you think that people in the news industry need to modify or expand their conception of “journalism” in order to account for the new and different ways that people can access and present information online?

Holovaty: People can define “journalism” however they’d like. At EveryBlock, what we’re interested in exploring is what sort of frequently updated information consumers want at the block level, and how they’d like to receive it. Whether this is called “journalism” or not is strictly academic. (I think it’s hard to argue against calling it “news,” though.)

I think people in the news industry should indeed modify their conception of what information they publish, and how they publish it. But should they modify their conception of “journalism”? Leave that to the people who have the time and inclination to debate semantics.

OJR: What has kept, or is still keeping, newspapers from having functionality like EveryBlock’s on their websites?

Holovaty: Unfortunately, there’s a lot. In the general case (and “general” means this excludes the newspapers out there who are doing great things online) –

* A lack of technical competence
* A culture so obsessed with daily deadlines that little thought/resources are put into paradigm changes
* A culture that disdains technology and science, particularly math, and, worse, actually takes pride in that
* Red tape
* Legacy systems
* Legacy attitudes
* People who ask “Is this journalism?”

OJR: How long can you keep the site running on your Knight funding? What happens after that runs out?

Holovaty: That remains to be seen! Knight has awarded a two-year grant, and we’re just over six months into it, so… ask again in about 18 months.

OJR: Who are you trying to reach with EveryBlock? How are your promoting the site?

Holovaty: We’re trying to reach residents of the EveryBlock cities. If you live in Chicago, New York or San Francisco, we hope to make your block’s page on EveryBlock into something that you’d find useful time and time again.

Something tells me you won’t be seeing EveryBlock billboards on the expressway, or EveryBlock ads on subway cars. That’s just not our style. We’ve e-mailed friends and family, and the rest has sort of happened through word of mouth, blogs and media coverage. This approach worked well for chicagocrime.org, which has (anecdotally) gained pretty good awareness over the past two and a half years here in Chicago, with zero traditional marketing on my part.

OJR: What’s the timeframe, and procedure, for expanding to other cities?

Holovaty: This is an interesting challenge that we knew going into the project: to some extent, the technology is scalable (i.e., replicable in a way that makes subsequent cities easier to launch), but at the same time, every city’s data is different. We still don’t know how that breaks down, resource-wise, but it’ll be fun to find out.

OJR: If a local publisher wants EveryBlock technology on his/her website, what should they do? Are you working with partners? Should they try to build this themselves?

Holovaty: We’re obligated under the terms of our Knight grant to release the site’s code under an open-source license at the end of the grant period. The idea there is to experiment and do some good for the news industry — that’s one of the core missions of the Knight News Challenge program (www.newschallenge.org), which is the contest I entered to receive this grant.

In the meantime, though, we’re very interested in working with partners — media companies, governments, bloggers and any other local-news publishers — in our EveryBlock cities. (Folks can contact us at feedback at everyblock.com.)

Personally, I wouldn’t recommend that news organizations attempt to build this themselves, but it’s obvious that I’m biased — both by having implemented EveryBlock and having worked at a number of news organizations.

OJR: What information, related to EveryBlock or not, do you really wish you could get in front of Web readers, but that you haven’t figured out either how to get your hands on or how to present effectively? (In other words, what’s the next challenge?) What’s it going to take to get that information out there?

Holovaty: That’s another good question. Regarding data on the current EveryBlock cities, I’d say we’re only at 10 percent of where we could be. We’re almost ready to add a couple of data sets that didn’t make it in time for launch, and we’re continually adding news sources and blogs to crawl.

One type of information that we purposefully haven’t included on EveryBlock is “static” information — the location of the nearest subway station, or the census demographics for your block. There’s been a small amount of user interest in this, but there’s a core difference in that type of information, namely that it’s not time-sensitive, and it would take some thinking to figure out how that fits in with our current “news feed for your block” paradigm. We’ll see what happens. It’s one of the many interesting problems we look forward to tackling.

That’s what drew more than 500 people today to the EconSM conference at the Beverly Hilton in Beverly Hills, California. Rafat Ali’s ContentNext Media put together the conference, which sought to explore the economics of social media.

Conference panels addressed the intersection of social media with marketing, movies, television, music, and, yes, the news. But as conference panelists expressed enthusiasm over this new era of public conversation, there remained few answers to questions about how news organizations ought to enlist social media to improve journalism… as well as their bottom lines.

“Follow the audience,” urged Vivian Schiller, vice president and general manager as NYTimes.com. “The more noise there is out there, the more need there will be for authoritative quality journalism.”

“All of these new platforms have opened new ways for us to get our journalism out and to start conversations with the public,” Schiller said.

But Topix CEO Rich Skrenta reminded that people cannot talk about stories that are not reported.

“There’s no paper in my hometown,” Skrenta said, citing a lack of local coverage for his Bay Area community from the two major newspapers in the area, the San Francisco Chronicle and the San Jose Mercury News.

“All sorts of shenanigans are going on, and nothing’s written about it anywhere,” Skrenta said.

“As the print classified dollars that used to pay for print journalism go away, how do you come up with a new model?”

Skrenta described how Topix, which is 80-percent owned by newspaper companies, recently reinvented itself as a social media company to pursue its goal of providing neighborhood-level coverage for every zip code in the United States.

As a news aggregator with a small staff, “we had to find a way to get the cost of obtaining this information down to zero,” Skrenta said. “We turned to social media.”

National Public Radio CEO Ken Stern warned against relying on the public to do journalists’ work.

“”A civil society depends on news, not just information,” Stern said. “But there’s a sense at many newspapers that we can get rid of reporters because we can get information from people off the streets for free.”

“The economics behind serious journalism are pretty brutal right now,” Stern said. “There is a demand for it, but the expense of producing that is immense. That’s a real serious challenge.”

The Wall Street Journal’s Kara Swisher suggested that journalists and readers can, and ought, to work together.

“The problem with journalists is that, at the beginning of blogging, they got all snotty about it. But there’s still a place for well-sourced news to go hand in hand with this,” Swisher said. “And I think the feedback system in the Internet helps protect coverage better than some people think.”

Swisher cited the microsite that she and the Journal’s Walter Mossberg have just launched as a model for news business faced with shrinking revenue.

“The cost of putting up our site is so low that if the WSJ online ad staff can’t sell to cover that, we should go out of business,” Swicher said.

Swisher cited conference organizer Rafat Ali’s paidContent.org as an example of a start-up that can report well-sourced news of high quality without racking up crushing expenses.

“If you can keep the [reporting] costs down, then it is is easier for the sales staff to make that nut they have to make,” she said.

Skrenta challenged newspapers to reevaluate long-standing business practices in an effort to better thrive online. As an example, he described the frustration some newspaper.com managers feel in seeing their staff’s content duplicated on other websites, some of which might rank higher in Google News and in search engine result pages, taking away readers – and revenue – as a result.

“Syndication made a lot of sense in print. But it doesn’t really make sense to do online,” Skrenta said. “Now is that Google’s fault, or maybe you should reconsider your online redistribution strategy?”

Swisher wrapped up the session on a positive note.

“People are hungry for this content. And that means there’s an opportunity for people with high quality standards to jump in here and provide something great.”

“If all the newspapers in America did not allow Google to steal their content for nothing, what would Google do, and how profitable would Google be?” the Los Angeles Times reported Zell saying to Stanford University class this week. Zell has agreed to takeover Tribune Co., which owns The Times.

Zell’s comment is as ill-informed as outgoing ASNE president Dave Zeeck’s were to his organization last week. I’ve been searching the Web through Google, and reading Google News, for years, and can’t recall Google publishing complete newspaper stories under the Google brand.

Yes, Google hyperlinks page titles and publishes short snippets of those pages’ content beneath them on its search engine result and Google News pages. Those links have helped millions of readers find newspaper stories that they would not have read otherwise.

Without newspaper content on Google News and in Google’s search engine result pages [SERPs], newspapers would face an even more dire future, as those millions of readers would find instead other, non-newspaper sources of news and information. And Google wouldn’t lose much at all. Google News doesn’t run ads. And I suspect that Google’s AdWords program would continue to haul in billions of dollars annually even without newspaper.coms in the SERPs.

Surely, not everyone in the newspaper industry shares Zell’s extreme view. For all the ignorance that certain newspaper managers exhibit in public forums, the newspaper industry employs many more sharp individuals with deep knowledge of how the Web works and how to make money from it. They’re found in the online departments of newspaper.coms and they deserve their chance to call the shots on how newspapers will approach the Internet.

The Zeeck and Zell attitude won’t save newspapers, and will serve only to further isolate them from a new and growing generation of Web-savvy readers.

I e-mailed several newspaper.com managers to ask them what they thought of Zell’s comment, and how they think the newspaper industry ought to approach the search engines.

Chris Jennewein
Vice President, Internet Operations, Union-Tribune Publishing Co.

I think newspapers should welcome search engines because they drive traffic to our sites. Our business on the Internet is all about building audience, and audiences find our sites through search engines. Fighting the new reality of Internet search is ultimately self-defeating for our industry.

Ken Sands
Online publisher, www.SpokesmanReview.com

Well, first reaction is that Zell is talking primarily about Google News, not Google itself. He suggests Google wouldn’t have anything without the content from media, and that’s just not true. Google News aggregates links. It doesn’t steal the content so much as organize it in a way that’s meaningful for lots of people. Mainstream media could have done the same thing but didn’t have the vision, the organizational capabilities or the technical skill.

Google argues that it’s a partner of the MSM, sending readers to each site (for free!). I can’t argue with that. Almost 40 percent of the visitors to our site come through Google searches. Those are typically one-time, one-story readers, but they do account for a huge amount of our traffic. There’s a legitimate question about whether these inflated numbers are meaningful… not sure this does our local advertisers much good.

On balance, I’d say Google does more good than harm. The Google Ad Sense for Newspapers alpha project is a good example. They’ve sent advertising to print newspapers for several months without taking a cut of the action. When the program goes to beta, they will begin taking a small percentage. They seem to understand that their best way of making money is to make sure that everyone else makes money, too.

Will Google some day dominate the media world? Quite possibly. But for now at least, it’s sure hard to pass up their goodies.

Steve Yelvington
Internet strategist, Morris Communications

Sam Zell’s comments make a lot of editors feel good, and editors need something to feel good about these days. But news search and aggregation are just a piddling part of Google’s portfolio of services. Google doesn’t even run advertising in its news channel, so claiming that newspaper content is behind Google’s profitability is just saber-rattling.

Google is a significant threat, but not because it’s “stealing” newspaper content. They’re rolling up a lot of local small business money that newspapers can and should be chasing. We’re not going to get that business by sitting around in our newspaper offices and wishing for the return of sideburns and bell-bottom trousers. We need to be out creating new solutions and coming up with better ideas.

I’ll post additional comments as they hit my in-box. Or you can post your thoughts through the comment button below.

Lasica’s piece got me thinking about ways to measure bias in search results. His observations became the basis for my recently completed master’s thesis, the findings of which may be of interest given the ongoing debate about the quality of Google News’s sources. While my analysis does not indicate an overall conservative or liberal slant, it does confirm Lasica’s suspicion that non-traditional news sources are injecting ideologically biased articles into Google News search results. The data show that articles returned in Google News searches are significantly more likely to have an ideological bias than those returned in searches on Yahoo News. (See below for detailed study results.)

The study examines search results for evidence of bias. By analyzing the content of articles returned in searches on the major-party presidential candidates in the days leading up to the 2004 election, it aims to assess the aggregator’s level of political bias. The study looks at balance within these articles as an indicator of bias, using results from the same searches on Yahoo News as a benchmark.

Notably, almost all of the additional bias in articles returned by Google News searches can be attributed to the site’s use of non-traditional news sources. In other words, if we consider only sources affiliated with old-media companies, the average bias scores for articles on Google News and Yahoo News are virtually identical.

Google News, still in beta three and a half years after its launch, tracks the top stories on some 4,500 English-language news sites, updating its index roughly every 15 minutes. The ability to effectively search this huge collection of timely information has helped make Google News one of the Internet’s most popular news portals, drawing about 5.9 million visitors a month.

But its groundbreaking method of identifying top stories based on how frequently they appear on sites in its index – and doing so entirely without human intervention – put the portal in critics’ crosshairs from the beginning.

That its algorithms are able to automatically determine relative importance of stories and present a front page with top stories in different subject areas has been interpreted by some as an ominous sign that computers will someday make human editors obsolete. At the same time, users have ridiculed bugs that cause the site to occasionally attach a photo to an unrelated article or elevate a relatively minor story to a prominent spot on its front page.

(It should be noted that my research concerns only the site’s search results and is unrelated to Google News’s practice of automatically ranking the top stories on its front page and section fronts.)

Because it uses no human editors, Google News has considered itself immune to bias.

“The algorithms do not understand which sources are right-leaning or left-leaning,” Google News inventor Krishna Bharat told Lasica last year. “They’re apolitical, which is good.”

But choosing which sites to index is perhaps as subjective an editorial decision as selecting the stories to play on the front page of a newspaper or website.

Google News does not share the list of sites it crawls, a practice that has resulted in a lot of speculation about its criteria for inclusion and the notion that there might be some ideological imbalance in its list of sources.

In an attempt to shed some light on the question, one blogger has written a script that grabs the news portal’s front page regularly and logs all the sources that it finds. The count stood at 2,256 as of Wednesday night, indicating that about half of the 4,500 sources have been identified.

Along with the mainstream sites in the list are a number of relatively obscure, online-only news sources (some of which are best described as weblogs), including the opinion sites MichNews.com and Useless-Knowledge.com.

Earlier this year, Google News dropped several sites, including the white supremacist journal National Vanguard, from its index after users complained that hate speech was turning up in searches.

It seems the news portal has been making plenty of its own news lately – albeit unwittingly. In March, Agence France-Presse charged in a lawsuit that Google was infringing its copyright by displaying AFP material on Google News pages. Days later, Google announced it would stop using AFP content. Since then, the Associated Press also has expressed “concern” about Google’s use of its material without payment.

And just this month we learned of a patent application filed by Google scientists in 2003, laying claim to methods of “improving the ranking of news articles” based on the “quality” of the articles’ sources – an apparent admission that relevance alone is not a satisfactory measure of an article’s value.

Google’s patent application offers the following variables, among others, as possible measures of a source’s quality: the volume of traffic it receives, the amount of content it produces, the speed at which it responds to breaking news, the size of its editorial staff and the number of bureaus it maintains. Any of these factors would appear to favor traditional media outlets.

If this is an admission that non-traditional sources are of lower quality, how does that square with Google News’s stated goal of increasing the diversity of viewpoints presented on its pages?

Google News currently does not distinguish opinion from fact in its search results (though it now attempts to identify press releases and satire). Hence, editorials and other opinion pieces frequently appear alongside straight news stories in search results. It is not clear that average users can make the distinction between the two, especially given the many online-only sources that peddle a confusing mixture of fact and opinion.

Ranking news stories based on some measure of quality may be a step in the right direction, but to maintain its credibility, Google News needs transparency – both in its selection criteria and its list of sources.

Key findings of the study

I was intrigued by the notion that a site without human editors might still be biased, and I wanted to test it scientifically. To do this, I analyzed the content of articles returned in searches on “George W. Bush” and “John Kerry” in the weeks leading up to the 2004 election. [More complete results and a detailed description of the research process are available in the full study (PDF).]

I wrote a crawler script to retrieve the results from Google News and Yahoo News for the search terms “George W. Bush” and “John Kerry” at four-hour intervals. The program run was during the two weeks preceding the Nov. 2 presidential election, resulting in a total of 80 “snapshots.” Each snapshot contained four sets of search results: “George W. Bush” on Google News, “George W. Bush” on Yahoo News, “John Kerry” on Google News and “John Kerry” on Yahoo News. The program also downloaded the full text of the top articles returned in each result list.

For each of five snapshots, chosen randomly, the first five articles from each of the four result lists were analyzed, ensuring an equal number of Bush and Kerry results and an equal number of Google News and Yahoo News results. This resulted in a sample of 100 articles, which then were examined sentence-by-sentence. Overall, 1,587 sentences were coded in one of five ways:

• Favorable to Bush
• Unfavorable to Bush
• Favorable to Kerry
• Unfavorable to Kerry
• Neutral

Using the values for each sentence, two scores are calculated for each article, measuring the degree of the article’s overall favorability to each candidate. These favorability scores could take values of –1 (completely unfavorable) to 1 (completely favorable), with 0 being neutral. For instance, a Kerry favorability score of –0.3 for an article would indicate that, on balance, 30% the content of an article is unfavorable to John Kerry.

Two charts – one for Google News and the other for Yahoo News – provide a basic summary of the data. They show the two candidates’ favorability scores for each article, plotted against each other. This facilitates comparison of the overall favorability of the two portals’ search results.

Favorability plots by news portal

Each data point represents an article, and its placement on the chart represents its favorability to the two candidates:

• Upper left quadrant: Article is favorable to Kerry and unfavorable to Bush
• Upper right quadrant: Article is favorable to both
• Lower right quadrant: Article is favorable to Bush and unfavorable to Kerry
• Lower left quadrant: Article is unfavorable to both

In other words, articles in the upper right and lower left are more balanced than those in the upper left and lower right. Articles closer to the center are more neutral. The circular boundary is a density ellipse drawn to make it easier to see patterns in the data.

To determine the direction of bias in a particular story, we compare favorability scores for Bush and Kerry. Where they are similar, the article is more balanced. Each article is assigned a balance score, which is the difference between the two favorability scores. A balance score greater than 0 would indicate bias toward Kerry while a negative score shows bias toward Bush. Both Google News and Yahoo News have average article balance scores that are very close to 0, indicating balanced search results. In other words, both the Google News and Yahoo News searches returned articles that were, on the whole, equally favorable to both George W. Bush and John Kerry. This is what we would expect to see of balanced search results at a time when public opinion was pretty evenly divided between the two candidates.

However, the spread of articles’ balance scores reveals an important difference: Articles returned by Google News tend to be significantly more biased in one direction or the other than articles from Yahoo News.

Besides being coded for favorability, articles were also classified by whether they came from an independent, online-only source (such as Salon.com) or a website affiliated with a traditional news source. A traditional news source is defined as a wire service, newspaper, magazine, TV station, radio station, broadcast network or cable network. (Content from one of these sources that is syndicated on a news aggregator such as Yahoo News is also considered traditional.) Of the articles returned by Google News, 40% were from non-traditional news sources, while only 24% of the Yahoo News results came from non-traditional sources.

When articles from non-traditional sources are omitted from the comparison, there is no significant difference in the spread of the article balance scores between Google News and Yahoo News. This indicates that virtually all of the difference in bias between articles returned by Google News and those returned by Yahoo News is attributable to Google’s use of non-traditional news sources.

AP’s most recent act of self-destruction was its April 18 announcement that it would start charging newspaper and broadcast clients an additional fee for using AP content on their web sites.

This move — sprung on its clients just as they are recognizing the urgent need to reinvent themselves in multi-media, web-driven modes — ignores powerful trends:

• All forms of content are migrating – each to its most appropriate medium. Readers and advertisers are following.
• As news media and other information providers jump into one media platform after another, the Web is emerging as their operational core.
• From blogs to open-source journalism to free newspapers, a wave of unpaid information is sweeping paid information off the media beach.
• As content loses value, expert editing and customer-driven bundling are becoming the tools for building audience. And audience — not content — is the news industry’s value proposition.

Contrast those trends with AP’s recent moves:

• Belatedly taking note of precipitous readership declines among young people, the AP is shopping around a youth publication prototype called APtitude. Its dominant story form is long narrative accompanied by a photo or two. But young people, as Rupert Murdoch recently pointed out, are digital natives, not digital immigrants. Their primary language is digital. When they do use their secondary language, print, their warmest response is to print formats that are highly visual and that are built with high proportions of short, non-narrative story forms. (See recent research at the Readership Institute.) This ill-conceived venture will add to the costs born by AP clients.

• Addicted to its transmission fee revenues, AP has chosen not to replace its high-cost distribution model (whose roots were planted in the telegraph era) with low-cost web distribution.
• Confronted with the rapidly growing need for web-specific content like Flash files, audio clips and other multimedia elements, AP has chosen to spend more of its members’ money to create that content rather than facilitate content-sharing among its members.

AP started as a cooperative. Today, it is a cooperative in name only. It’s time to take a lesson from music swappers and invent the new AP – a digital cooperative, a Napsterized news service.

The 21st Century news business needs a peer-to-peer network that lets local operations drive cost out of their non-local news packages, divert resources to local web content creation and operate on a level playing field with bloggers, citizen journalists and internet pure plays.

The network should be a closed, password-protected system. All content would live on members’ computers and would be indexed and shared through a central search. Open source software would keep costs down and assure compatibility with both Mac and Windows PCs.

Sharing would be governed by a karmic balance. The more you make available to the network, the more you can take out. An organization in karmic deficit would have to true up by paying a surcharge on the monthly fee.

An elected committee would administer the network, set sharing rules and levy the monthly fees – which primarily would pay for technology.

The network should support subgroups, allowing operations under common ownership to share files within the larger system and make those files available outside each subgroup as they see fit. This sub-group ability also would encourage regional networks — or even groups with a special interest in a particular story or subject area — to form ad hoc.

Members would have to adopt thorough formatting taxonomy and keywording schemes that would make articles easy to search, sort and parse for publication. Suitable schemes already exist through independent standards bodies such as the International Press Telecommunications Council and the news division of the Special Libraries Association.

A PubSub-like function would allow a member to be notified when stories with key topics hit the networks. For instance, a Knoxville newspaper or broadcast outlet would get an alert when any member uploaded a story about the Great Smoky Mountain National Park.

Of course, editing standards would be as varied as the members – and in some cases would not be up to the AP’s standards. But most news operations – particularly those in small or mid-sized markets – use less wire copy these days and try to localize what they do use. So long as members attribute with care, journalistic standards will not be in jeopardy.

If the network pulled in one or two large U.S. news organizations plus a few from abroad, national and world news demands could be met easily. Members with adequate editing capacity could work network content into tight national and world packages and make those available – perhaps for added Karma credit within the network.

The AP creates very little exclusive coverage. With enough members and shared editing capacity, the nation/world category would be dealt with easily.

Perhaps the toughest content area to cover would be statistical services like sports agate and stock tables. But think about that for a minute. Are stock tables still relevant when every investor has her portfolio set up on a financial web site? And couldn’t a committee of sports editors come up with an alternative source of box scores?

Although the technological challenges of Napsterized news might seem formidable to many news people, they are, in fact, minor. Most of the technology already exists, much of it is in open source and dealing with it isn’t rocket science.

Most news organizations already use the Internet extensively, have plenty of file servers and understand Windows/Apple networks. There would be no massive, centralized technology. The concept is lean, with most of the computing power residing at each member’s location.

As we started talking about this, we asked ourselves, “Yeah, but when have newspapers ever succeeded in working together? Look at New Century Network and all the other cooperative brainstorms that failed.”

But all such initiatives started with fatal flaws:

• Some took control away from participants. Our idea leaves control with the members.

• Ego wrecked many of them. But egos tend to calm down when no power position exists. Sharing is just sharing.
• Voracious money pits swallowed most of them. This idea, to the contrary, could save news organizations a lot of money. Imagine driving 90% of the cost out of a newspapers’ wire service budget line. How much excellent local coverage could be created with the money saved?

If AP had its collective head firmly inside the 21st Century, it already would be moving at least parts of its services in the Napster direction. But AP is like any business confronted with a disruptive technology. Its first inclination is self-preservation, not cannibalization.

One of the smaller news services with less to lose could jump into Napsterized news, but the small ones tend to follow the lead of the big ones.

The best bet is a start-up consortium, perhaps starting with a few of the smaller corporate groups and independent newspapers.

We’re ready to host the first meeting. Anybody want to talk?

Bob Benz is general manager of print web operations for E.W. Scripps. Mike Phillips is the company’s newspaper division editorial director.

Nothing could be weirder than the first time you watch CNN’s “Inside the Blogs,” a segment on Judy Woodruff’s “Inside Politics.” CNN’s “blog reporter” Jacki Schechner does a tag team with producer Abbi Tatton, as they bring up Weblogs on dueling computer monitors and tell you what the blogs are saying about the issue du jour.

But this is the logical next step in the evolution of blogs — whether the old-school bloggers like it or not — as mainstream culture grapples with the concept of Weblogs and their growing power in punditry. Increasingly, mainstream media outlets will need to explain how memes spread in the blogosphere, who is saying what, and what their agenda is, if any.

While bloggers might resist attempts to superficially or technologically assess “what the blogosphere thinks” on a particular issue, these aggregated looks by CNN and now Slate’s “Today’s Blogs” (also launched in February) are windows into a world that is new to so many people. Plus, very few bloggers would turn down the promotion they receive from mainstream media exposure.

And for journalists, aggregators can be a blessing, giving them a quick read on an issue, offering new insight on a topic of research, or putting them in touch with new sources for a story. After journalists’ stories have been posted online, they can track feedback by searching via engines such as Technorati or Daypop.

So here’s a guide to some of the better blog aggregation efforts so far. I’ve kept away from pure RSS readers such as NewsGator or Bloglines, as they offer more of a software solution than an editorial or weighted view into blogs. Keep in mind that most of these features and services are new and sometimes buggy or inaccurate. But having a flawed reading is better than the alternative: reading all 7 million blogs yourself.

The list is in order of those with the most human touch to those that employ the most automation. In all cases, I tried to focus on how blogs viewed the recent death of Pope John Paul II.

CNN’s Inside the Blogs

So how do you do good TV about blogs? Many cable news shows have started booking bloggers such as Robert Cox and Jeff Jarvis regularly. But CNN decided to take a different tack with its blog reporter and politics producer going on-screen in a special section of the set. The pair shows actual blog pages and postings with the CNN talking heads poking out of a small box in the corner of the screen. On the pope’s death, they noted that some bloggers were actually happy about it (though they didn’t name names), and pointed to The Pope Blog and the Catholic Insider podcast as good sources.

The broadcast has shades of the old TechTV channel, though the segment was severely hampered by not even listing the blog URLs — nor were they even on the show’s Web site or transcript. Predictably, some bloggers cried foul for bias, and one knocked CNN for referring to many more conservative blogs than liberal ones. Still, the segment is worthwhile despite its shortcomings, even if it only helps Woodruff and other newbies “get” the blogosphere. Of course, the special guest on “Inside Politics” that day, Rev. David O’Connell, president of Catholic University, couldn’t help getting off a dig about the “incivility of blogs” that didn’t respect the pope.

Grade: A for effort, C+ for execution

Today’s Blogs

Slate has made its mark not just with political reporting and insightful features, but with its people-powered aggregators like the groundbreaking Today’s Papers and In Other Magazines. The concept is simple: Have a writer read other media and sum it up in a witty way for the time-challenged consumer. Taking on Weblogs makes a lot of sense for Slate, and they promise in their promos to deliver “Five Million Blogs in Five Minutes.”

Today’s Blogs is a lively read and largely focuses on the most popular A-list bloggers and the memes they are following. The feature, written by rotating interns and editorial assistants, has the disadvantage of trying to sum up millions of blogs in the space of a short column — unlike Today’s Papers’ limited purview of the top stories in the top five newspapers. Worse still is the ethical mire the column descends into by almost always linking to news stories in the Washington Post, the flagship paper of Slate’s new owner. There are plenty of resources such as Daypop Top 40 and Blogdex that can help point the Slate interns to which news stories are most linked in the blogosphere; these should be the ones that get the links from Today’s Blogs.

[Update, 4/6: Slate editor Jacob Weisberg said: "We do NOT have a policy of favoring the Washington Post for newslinks in the body of stories. Our policy is to use whatever story seems best to us, or whichever one has more relevance to the bloggers we're covering. Apparently, the editorial assistants have been favoring the Post for just the last week. This was not according to our editorial policy, and they won't be doing it that way any more."]

Grade: C+ for effort, B+ for execution

CJR Daily’s Blog Report

From the ashes of CJR’s much needed CampaignDesk.org site during the election comes CJR Daily, a continuation of that site with an expanded focus on daily journalism. One of the regular features has been Blog Report, which also has its roots in election coverage and politics. CJR Daily’s managing editor Steve Lovelady has taken heat for saying, “The salivating morons who make up the lynch mob prevail,” in reference to bloggers taking down former CNN honcho Eason Jordan.

But that hasn’t stopped Blog Report from being a smart read on chatter in the blogosphere. Sometimes CJR’s rotating writers take on a more patrician tone toward those unruly bloggers, but other times they get into the spirit of things. On the day of the pope’s viewing at the Vatican, for example, Blog Report was agog over the start of the baseball season and how bloggers reacted to the Yankees’ win over the Red Sox. In their defense, though, CJR Daily had a number of other items on papal coverage.

Grade: A- for effort, B+ for execution

Taegan Goddard’s Political Wire

Goddard, an author and political consultant, runs one of the more interesting hybrid aggregator sites, Political Wire. The Front Page is a blog that Goddard maintains with an editorial voice, looking mainly at mainstream news reports on U.S. politics. Then he has two separate automated sections for liberals (“Southpaws”) and conservatives (“Wingers”). Again, the focus is on established A-list bloggers such as Daily Kos and Powerline, but it often has the look of a high-powered group blog, with a clean layout that doesn’t force you to click through to the blog posts themselves. On the pope, Goddard had a nice pointer to “The Shoes of the Fisherman,” an old novel and movie about a Russian pope that foreshadows in many ways the rise of Pope John Paul II. Political Wire does a better job of making sure you don’t miss the top political issues without digging much past the A-list.

Grade: B- for effort, A- for execution

Memeorandum

While programmer Gabe Rivera’s Memeorandum is a fully automated look at the top news stories and blog posts relating to those stories, there is a very human element in Rivera’s choice of which Weblogs to aggregate. While Google News has also made the claim of full automation, we know that some person had to program the software, pick the sources and rank them. Rivera was aggregating in “the low hundreds” of blogs as of last summer, and he mainly focuses on top U.S. news stories and controversial op/ed columns that catch fire in the blogosphere. Rivera maintains that stories are chosen by popularity of links in to them, as are Weblogs, similar to the Technorati system. The page is simple and clean, but the headlines and blurbs don’t always give you an idea of the subject matter. For example, an item on Peter Jennings’ lung cancer originally linked to his bio instead of an AP news story.

Grade: B for effort, A for execution

Kinja

Though it hasn’t received as much attention as Nick Denton’s other blog ventures, Kinja could evolve into a valuable blog aggregator in time. Currently in beta, Kinja allows you to follow various favorite blogs on its site without encumbering you with the nuances of RSS or reader software. As for aggregation, there’s the Editor’s Digests of subjects such as Baseball, Food, Sex and Politics (with Liberal and Conservative subgroupings). A strength of the site is the simple navigation and nice layout for newbies, but its weaknesses are pretty glaring. Technically, headlines spew out in unintelligible bites at times, as do the blurbs from blog posts; editorially, some blogs were miscategorized (e.g. Outside the Beltway in the Liberal section); ethically, you can’t help but sneer at the prominent placement of Denton blogs Gawker, Wonkette, et al. However, in the Editor’s Showcase of good blog writing, there was a recent inclusion of LA.comfidential, a competitor to Denton’s Debaser blog. With some technical attention and a more neutral viewpoint, Kinja could become indispensable.

Grade: C for effort, D for execution, B for potential

Findory

This comes in as perhaps the most high-concept entry. Findory attempts to take the Google News format two steps further — by offering an interface with just news headlines and blurbs, and another with just blogs. But Findory’s most interesting feature is that the weight and placement of stories depends on what you click through to read on repeat visits. There’s even a little icon that comes up next to stories it considers to be of keen interest to you. I decided to have a little fun by clicking on the weirdest news stories relating to aliens and a woman breastfeeding tiger cubs in Myanmar. Sure enough, on future visits, I was presented with more of those types of stories. Findory makes you think before you click through (“will I be permanently tainting future recommendations?”), but still provides you with top world stories despite any fetish you might have for less newsy items. It has a very simple design that could use photos to break it up. And blog posts are sometimes miscategorized by the blogger’s background rather than the subject matter. Otherwise, a fascinating experiment.

Grade: A- for effort, B+ for execution

Which are your favorite blog aggregators, human or automated? Click the button to tell us which ones you like and why.

For the past year, I’ve wanted to peek inside the operation at Yahoo News and see for myself just how they made the sausage. The problem is that Yahoo is the quintessential Silicon Valley company, paranoid about losing its trade secrets or embarrassed that a reporter might be flummoxed that there’s no “there” there.

My plan to visit during the 2004 Summer Olympics was squelched. My plan to visit on U.S. Election Night was squashed. Finally, the opening came when Yahoo News was readying a redesign of the site, its first since 2002. I was invited to sit in on a production meeting related to the redesign — due to launch in public beta today, but now slated for later this month.

Yahoo News lives in Building F on the Yahoo campus, located at the end of a row of shiny modern office parks along Mathilda Avenue in Sunnyvale, Calif. There are security guards stationed at each parking lot, and you must sign a waiver when you check in at the building’s lobby saying you won’t steal trade secrets or photograph anything without written permission. I clumsily went to the wrong entrance of the building, and some employees let me in the locked door as they went out. So much for security.

The office is what you’d expect at a dot-com success story, with brightly painted walls and well-lit, airy spaces. The conference room was named “Gilligan,” and it had purple carpeting and purple seats with yellow piping — the official company colors. Product managers sat alongside engineers and designers and went over the various bugs in the Yahoo News redesign. As an outsider, I could barely follow the stream of technical jargon and nicknames for features.

Someone suggested they “tweak the mocks.” A problem arose over “bread crumbs” being shown. A discussion ensued over renaming the Op/Ed section Opinions or Commentary. On my first look at this redesign, it seemed cleaner and less cluttered — with tabbed navigation at the top instead of the side. (You can see similar tabbed navigation at Yahoo News Asia already.)

Later, Yahoo News product manager Jeff Birkeland gave me a tour of the redesign — forget the building. A new “toggle” feature lets you view the top headlines from each news source under each category (Business, Entertainment, Sports, etc.). But one tab was titled “My Sources,” and that was the magic tab, bringing up your selection of RSS feeds, automatically populated by your RSS feeds chosen from My Yahoo, if you kept a My Yahoo page.

Suddenly, Yahoo News was more than just a collection of licensed content from established news sources — it was every news source and Weblog that had an RSS feed. Personalization had landed at the front door of Yahoo News, after making such a huge splash at the front door of Yahoo itself.

And just as the RSS technology from My Yahoo was being injected into Yahoo News, a beta search technology called Y!Q was also coming to Yahoo News. As you can see in these more crude examples, a highlighted term within a news story brings up a little pop-up box with relevant links on the subject within Yahoo’s own pages and from Web pages outside Yahoo.

Mixing automation with the human touch

Yahoo sits at the intersection of technology and media, fueled by the Internet boom in advertising and paid content and led by a Hollywood studio executive in Terry Semel. Plans are afoot to move most of the editorial operations to the new Yahoo Media Center in Santa Monica, Calif. Though Yahoo spokesmen are relatively tight-lipped about who’s moving when, signs abound that new editorial hires will be heading south. Just today came word through PaidContent.org that Yahoo had hired MSN general manager Scott Moore from Microsoft.

Another of the recent hires was Neil Budde, an online news veteran and the founding editor and publisher of the Wall Street Journal Online. Budde came on last November as the new director of news at Yahoo, taking on editorial and business oversight. Budde told me in a phone interview that he doesn’t find Yahoo News all that different from WSJ.com. And he considered the best difference to be the huge number of engineering resources at Yahoo.

“I think it makes it a lot easier,” Budde said. “One of the reasons I came here is there are so many resources to draw on for all the technology. Plus, with a network like Yahoo, you have so many insights on how people consume news. What’s been great is you have the ability to tie into personalization that’s already been built in My Yahoo, and it allows you to bring in the search capabilities into news, like the Y!Q capabilities. … It’s different than at a traditional news organization where some of those things are harder to build or you’re building it from the ground up.”

While not getting into too many specifics on how the editorial process works — or how many people work at Yahoo News (likely in the dozens and not hundreds) — Budde told me that the team uses a combination of automation and direct human input. There’s a team running Full Coverage, which packages Yahoo content with links to outside news sources and resources on the Web. There’s a team for Sports, for Finance and for the content deal with SBC. There isn’t a 24-hour news team, though many editors have pagers that will go off in case of a breaking story.

“My [past] experience is with more people doing things and less automation, but the folks who have built Yahoo News up ’til this point have done a great job of being very smart about where they can automate and how much they can automate,” Budde said. “That frees up the human editors to do what human editors should do, which is make editorial judgments.”

Budde explained that Yahoo’s longtime partners such as the Associated Press and Reuters have helped streamline the automation process by working closely together with Yahoo News. “Going back many years, when online news was first developing, Yahoo News was one of the first to educate people like Reuters on what’s the best way to build their feeds for online products as automatically as possible,” he said. “So you can be smart about having the human editors do intelligent work on top of that.”

If you assume that the most trafficked Web pages are Yahoo.com, MSN.com and AOL’s proprietary home page, then you can deduce that news headlines on those home pages will generate the lion’s share of traffic to those sites’ news pages. Budde wouldn’t break down exact numbers on originating traffic for Yahoo News but said Yahoo.com, Yahoo search, and My Yahoo are key drivers.

According to Nielsen/NetRatings, Yahoo News is the second most trafficked News & Information site on the Web — and top Current Events/Global News site — with 20.8 million unique visitors in February 2005. Over the past two years, the site’s traffic has hit peaks and valleys depending on news stories such as U.S. elections and the Southeast Asian tsunami. But generally, traffic has gone from 16-19 million uniques in 2003 to the low 20 millions in 2004, while time spent on the site has gone from the high 20 minutes to low 30s.

One way to build on the time spent on the site is to make the story pages more rich with links to relevant content on other pages, according to Budde. He noted there’s a shift under way for news sites to treat story pages as entry ways for so many people entering via searches.

“People don’t start from the front page, they often start at story pages,” Budde said. “So if this is the first point of entry into our site, what can we do to expose people there to more of what we have available? I think it’s important for any news site. [We want to make] that the central focus, as opposed to so many editors who want to focus on the front page because that’s what the traditional editorial role has been — the front page.”

Taking RSS to the masses

While Really Simple Syndication (RSS) technology hasn’t really caught fire in the public consciousness, Yahoo has integrated RSS feeds into its personalized offering, My Yahoo, without really having to explain it. You just add content to your page by topic or by Editor’s Picks — of course, heavily weighted with Yahoo-hosted content.

That integration, started in January 2004, was just one of many announcements by so many online media outlets that have launched RSS feeds or a branded RSS reader. But this move by Yahoo had huge implications, because more than anything else related to RSS, this really did bring RSS to the masses.

“[Yahoo's] integration of RSS into My Yahoo and Yahoo generally is so profound in terms of what it means for Yahoo,” said John Battelle, author of the upcoming book, “The Search” and founder of the Industry Standard magazine. “Yahoo News, in the Web 1.0 way, was supposed to be your RSS aggregator. Now, they still do Yahoo News, but they’ve given the reins over to users to do feeds that aren’t necessarily blessed by Yahoo.”

So now those feeds have spread from My Yahoo to a mobile service, a desktop ticker and to Yahoo News — probably the most appropriate place for them. Birkeland, the product manager for news at Yahoo, said that they have to perform a balancing act on giving people the content they want without giving them too much to handle.

“You could give people a lot of content, but we want to avoid what we call ‘firehosing’ people,” Birkeland said. “We don’t want to point the firehose at people and let them figure out what’s going on.” But he’s excited about giving people RSS feeds at Yahoo News, which “literally will allow your own choices to exist right in the middle of our front page.”

Still, there are limits. While you can put your feeds into news sections, you still can’t pre-empt the top story of the day. Birkeland says it’s important that Yahoo News still has enough control to convey important breaking news and allow serendipity — so people might learn things from unfamiliar sources and not just from an echo chamber.

While Yahoo News does have its own smallish newsroom in Building F, the operation draws on technology and resources from around the company. Scott Gatz, senior director of personalization products at Yahoo, has become known as “The RSS Guy,” bringing feeds to various Yahoo initiatives, including a search engine for RSS feeds, and the Yahoo News redesign. He told me the key to success for winning over the masses was not bothering to even call them “RSS feeds.”

“If you look at how we’ve integrated RSS into Yahoo News, we’re not actually using those three letters very much,” Gatz said. “So you look at it, and it says what would you like to add to your political news, here are some political blogs. Would you like to add CNN or MSNBC onto your news page? The fact that it happens in XML or RSS isn’t the important thing. Most of the users don’t want to have to figure that out.”

Skirting legal issues

Gatz says that Yahoo has more people using RSS than any other service, a number of users that’s “in the low millions.” He estimates there are 6 million RSS feeds around the Web. Bloggers take note: When people go to choose RSS feeds at My Yahoo or within Yahoo News, the Yahoo editors suggest various sources, from mainstream news sites to niche content to blogs. Getting into those slots likely will draw more traffic than from any other feed source.

Battelle, who also manages the popular BoingBoing blog, says that he has a contractual deal with Yahoo — with no money changing hands — where Yahoo lists BoingBoing as a feed to add, and BoingBoing has an “Add to My Yahoo” button on its blog. Those buttons are sprouting up like wildflowers on popular blogs now. But Yahoo allows its users to add any feed they want easily, and puts the RSS search engine right on those pages.

While Google was recently sued by Agence France Presse for listing its content in Google News without licensing it, Yahoo News has a deal with AFP and isn’t worried about causing trouble by adding RSS feeds to the mix. Why? Because any source that includes RSS feeds would want to have as many people add them as possible. Just having RSS feeds alone implies an invitation to run headlines on a news reader, Birkeland said.

“There is a clarifying effect of money, that Yahoo is willing to employ [with AFP and others],” Battelle said. “We’re going to syndicate your stuff, and therefore we’re going to pay you for it. And if you have a free RSS feed, we’re going to allow anyone to add it to it.”

While the BoingBoing deal doesn’t include money for placement, that deal is only short term. In the long run, Yahoo might be able to monetize where it places RSS feeds for users to add, just as it eventually charged money for e-mail services, personal ads and personal home pages. Just the fact that Yahoo editors are making a directory of RSS feeds and picking out the best blogs could eventually make it the de facto place for finding and sorting Weblog and niche content.

While a million RSS readers and startup companies have elbowed each other for attention, Yahoo has quietly become the elephant in the room for news feeds.

“We’re the first portal to do anything with RSS, and we’ve now had a year and a half to refine and improve it,” Gatz said. “The ultimate goal was to bring it to the masses, and now that we’ve learned that and made it easy, how do we extend it across the Yahoo network? The concept is powerful, that anyone can subscribe to anything.”

While Yahoo News has a long history of filtering and aggregating content on the Web, the future holds even more promise, with Moore and Budde on board, the resources of a forward-thinking technology company behind them and the new synergized headquarters in Santa Monica. The key will be their balancing act of human editing and automation, of collating content and helping millions understand what’s important in the news today.

* * *

The New Yahoo News
A rundown of some of the new features of the redesign, due for public beta later in April:

• “Toggle” feature lets you see news headlines from a particular news source at a glance.

• “My Sources” tab on each section lets you see your RSS headlines — and you can add feeds to the page.
• Tabbed navigation on top of each page, with more weight and space given to the top story package.
• Y!Q technology embedded into key words in stories brings pop-up box with links to relevant Yahoo and Web pages.
• Story pages will have one square ad within story instead of a top and side banner.
• “IM story” option lets you send a news story via Yahoo Messenger instant messaging service.

In 2003, social networking sites Friendster, tribe.net and LinkedIn started business. Now there are up to 200 social networking sites covering everything from business contacts to dating. In New York and Boston mobile service Dodgeball informs users when a friend of a friend (FOAF) is within 10 blocks. FOAF is now also the name of web-based software protocol that describes people and their friends.

Eurekster is a search engine that matches your search terms with pages viewed by your friends and FOAFs. In December Netflix launched a new tool, where members can see recommendations from their friends. Delicious Library, a hugely popular book, CD and DVD inventory management program for the Mac will include social networking tools in the next version so friends can share their tastes.

The concepts, then, are all around the Net, but what are social networks, really? How do they work, and what’s their relevance to the news business?

There are two flavours of social network: the broad and the deep. Broad social networks are any form of on-going interaction between people, and they have existed since the emergence of tribes in human evolution.

“There’s a whole spectrum of relationships,” said Howard Rheingold, who in 1993 wrote a book about social networks and virtual communities. “For example, somebody in the store, whose name you don’t know but from whom you might buy regularly, they are part of your social network, the person who delivers your mail, your friends and family. There are all sorts of degrees of relationships within social networks.”

This sociological definition comprises a broad matrix of relationships that feeds one of the most powerful appetites in people: the hunger to connect with others through a common goal, shared interests or mutual benefit.

The deep definition refers to the increasingly common online practice of developing tools to leverage the friends of a friend for mutual benefit. “I think the use of the term social networks has changed. Since Friendster, I think it’s come to mean services like Friendster. Of course, the term pre-dates the Internet,” said Rheingold.

On the Internet, social networks are old news. In the early days of the Arpanet, originally designed for military communication, designers saw the new network as a means of collaboration between researchers. Collaboration was one of the ideas that inspired Tim Berners-Lee to develop the World Wide Web.

“Social Networks, codified or not, provide a mechanism for prioritization and filtering of information, including news,” said Tony Gentile in an e-mail interview. Gentile, a strategic marketing consultant for Internet companies, runs buzzhit.com, where he writes about emerging technologies and trends.

Technologies, like bulletin boards, Internet forums, e-mail, chat, and instant messaging are all social networking tools. Amazon recommendations, its listmania, reader’s reviews and ‘people who bought …’ listings all serve as relevant context and timely information in a world of virtually unlimited choice. They are simple and basic functions that tap into low level, or broad, networking concepts.

Deeper social networking is emerging with weblogs, where people start a conversation and, often, invite others to comment. Add trackback into the mix and you get the formation of conversations, relationships and networks. Sites like LiveJournal exist to enable blog networks. Rathergate, Easongate and the Tsunami disaster are all examples of how elaborate social networks develop spontaneously around the news, with people sharing news, evidence and analysis.

“In the case of blogs and feeds, your social network may do more than simply refer content to you; they may annotate it with their own analysis and commentary,” said Buzzhit’s Gentile. “This combination of social networking, blogging, referral and annotation is at the heart of service likes Rojo.”

Similarly, wikis such as Wikipedia and open source software development, like Linux or Firefox, draw together a network of disparate individuals into a shared goal.

Again, all this is old news, but now companies want to develop ever more elaborate social networking components into their software and services. The Netflix and planned Delicious Library tools are significant advances on Amazon’s ratings, for example.

So far, most newspapers have done little to develop the tools required to enable interesting, broad networks needed to foster ongoing relationships with their readers. Most newspaper innovation stopped with ‘e-mail this story’ and subscription models. Even less exists for deep networks.

“I know of very few major U.S. newspaper sites that allow readers to comment on articles. Most channel interaction is through ‘Letter to the Editor’ from posts or e-mail addresses,” said Buzzhit’s Gentile. “And that’s a big issue. Providing the tools is the easy part. Encouraging usage, that is, building community, starting a conversation, is far harder. So, yes, newspapers need to provide the tools … but fundamentally, they must reinvent their relationship with their readers.”

But now some newspapers, too, are beginning to deploy social networking tools and components into their core services. The tribe.net experiment by The Washington Post and Knight-Ridder is an attempt to shore up the rapidly disappearing newspaper classified market and to engage more meaningfully with readers.

In the UK, Silicon.com, like other news sites, encourages reader comments on every story. Those comments sometimes generate stories in their own right. The UK’s Guardian uses discussion groups and regularly invites readers to chat and pose questions to newsmakers and journalists on its live talk section. Additionally, it has a lively set of message boards where readers can exchange views.

Salon.com offers its own blogging service. In citizen journalism, sites like the Northwest Voice develop networks around local news, combining aspects of a blog and a wiki. Across Netspace newspapers are beginning to deploy social networking tools that connect readers more directly to the paper and each other.

At Yahoo News, for example, the ‘Most Popular’ section gets the greatest number of visitors. Yahoo doesn’t break down the numbers to the ‘most popular’ page, but the News site overall received 23 million visitors in January.

“We have three flavours of ‘most popular’: most e-mailed, most viewed and highest rated, and also we’re looking at ‘most searched,’ and we may find ways to start using that data on the site as well,” said Neil Budde, Yahoo’s Director of News. “With over 20 million visitors each month it provides information about what’s interesting to a wide swath of people.”

Right now Budde wants to look at ways social networking could enhance traffic data and reader experience.

“What we would look for down the road is ordering stories more dynamically, based on not just an editor’s view or a mass view, but maybe take social networking, like ordering stories based on a reader’s group of friends, or on people who have similar interests to your own,” he said. He thinks similar tools could be developed around readers’ comments.

But are social networking tools worth all the fuss? Will it have an impact?

“Are you asking me if there’s a way to make money off of a cost-effectively, virally acquired audience of above-average participatory users who produce metadata based on their activities? I think the answer is clear ,” said Buzzhit’s Gentile. “… (Benefits are) audience growth, participation and metadata. All of which should ultimately improve monetization.”

That’s worth pondering. In January the New York Times carried a story by Eric Dash about how the Dow Jones’ purchase of CBS Market Watch was prompted by concerns over growing ad revenues: The online WSJ was running out of space for the ads. Even though the WSJ can add an infinite number of pages to its site, advertisers only like the popular ones. The thriving connection with readers outlined by Gentile is one way of maximizing the value of the pages newspapers already have.

Still, not all are convinced that news is an exciting proposition for social networking.

“The question is … is it already too late? Content distribution is gravitating toward feeds, and feed readers are integrating social networking. Newspaper sites might be able to integrate SN via FOAF, or similar open frameworks, but the likelihood of a consumer inviting 30 friends to a newspaper site seems … remote,” said Gentile.

Right now, I have a feeling Gentile is right. ‘People who read this story also read … ‘. It doesn’t work for me. But there’s no reason why it shouldn’t.

Every subject under the sun has a history; likewise every subject under the sun has news. For dating, it’s the hot new singles bar; for cinema, it’s the latest releases and their reviews. Can’t newspapers develop as a node that taps into people’s desire to network, by sharing interests and information across all topics? Can newspapers open their pages to readers and seed the conversation with content they already produce? (see sidebar, some imaginative speculation).

Whether newspapers can effectively deploy social network technologies, and what effect they may have, are moot points. But according to Yahoo News’ Budde, one thing is sure:

“Social networks are going to continue to evolve, and all the media need to pay attention to it.”