Inside Yahoo News: Aggregator brings RSS to the masses

There are two ways to view Yahoo News. One is to dismiss it as simply a collection of other people’s journalism, slapped together and considered just another feature of a big Internet portal. The other is to sit in awe of a site that includes some of the best journalism created, packages it in a simple way with links to outside sources and balances human judgment with technological innovation.

For the past year, I’ve wanted to peek inside the operation at Yahoo News and see for myself just how they made the sausage. The problem is that Yahoo is the quintessential Silicon Valley company, paranoid about losing its trade secrets or embarrassed that a reporter might be flummoxed that there’s no “there” there.

My plan to visit during the 2004 Summer Olympics was squelched. My plan to visit on U.S. Election Night was squashed. Finally, the opening came when Yahoo News was readying a redesign of the site, its first since 2002. I was invited to sit in on a production meeting related to the redesign — due to launch in public beta today, but now slated for later this month.

Yahoo News lives in Building F on the Yahoo campus, located at the end of a row of shiny modern office parks along Mathilda Avenue in Sunnyvale, Calif. There are security guards stationed at each parking lot, and you must sign a waiver when you check in at the building’s lobby saying you won’t steal trade secrets or photograph anything without written permission. I clumsily went to the wrong entrance of the building, and some employees let me in the locked door as they went out. So much for security.

The office is what you’d expect at a dot-com success story, with brightly painted walls and well-lit, airy spaces. The conference room was named “Gilligan,” and it had purple carpeting and purple seats with yellow piping — the official company colors. Product managers sat alongside engineers and designers and went over the various bugs in the Yahoo News redesign. As an outsider, I could barely follow the stream of technical jargon and nicknames for features.

Someone suggested they “tweak the mocks.” A problem arose over “bread crumbs” being shown. A discussion ensued over renaming the Op/Ed section Opinions or Commentary. On my first look at this redesign, it seemed cleaner and less cluttered — with tabbed navigation at the top instead of the side. (You can see similar tabbed navigation at Yahoo News Asia already.)

Later, Yahoo News product manager Jeff Birkeland gave me a tour of the redesign — forget the building. A new “toggle” feature lets you view the top headlines from each news source under each category (Business, Entertainment, Sports, etc.). But one tab was titled “My Sources,” and that was the magic tab, bringing up your selection of RSS feeds, automatically populated by your RSS feeds chosen from My Yahoo, if you kept a My Yahoo page.

Suddenly, Yahoo News was more than just a collection of licensed content from established news sources — it was every news source and Weblog that had an RSS feed. Personalization had landed at the front door of Yahoo News, after making such a huge splash at the front door of Yahoo itself.

And just as the RSS technology from My Yahoo was being injected into Yahoo News, a beta search technology called Y!Q was also coming to Yahoo News. As you can see in these more crude examples, a highlighted term within a news story brings up a little pop-up box with relevant links on the subject within Yahoo’s own pages and from Web pages outside Yahoo.

Mixing automation with the human touch

Yahoo sits at the intersection of technology and media, fueled by the Internet boom in advertising and paid content and led by a Hollywood studio executive in Terry Semel. Plans are afoot to move most of the editorial operations to the new Yahoo Media Center in Santa Monica, Calif. Though Yahoo spokesmen are relatively tight-lipped about who’s moving when, signs abound that new editorial hires will be heading south. Just today came word through PaidContent.org that Yahoo had hired MSN general manager Scott Moore from Microsoft.

Another of the recent hires was Neil Budde, an online news veteran and the founding editor and publisher of the Wall Street Journal Online. Budde came on last November as the new director of news at Yahoo, taking on editorial and business oversight. Budde told me in a phone interview that he doesn’t find Yahoo News all that different from WSJ.com. And he considered the best difference to be the huge number of engineering resources at Yahoo.

“I think it makes it a lot easier,” Budde said. “One of the reasons I came here is there are so many resources to draw on for all the technology. Plus, with a network like Yahoo, you have so many insights on how people consume news. What’s been great is you have the ability to tie into personalization that’s already been built in My Yahoo, and it allows you to bring in the search capabilities into news, like the Y!Q capabilities. … It’s different than at a traditional news organization where some of those things are harder to build or you’re building it from the ground up.”

While not getting into too many specifics on how the editorial process works — or how many people work at Yahoo News (likely in the dozens and not hundreds) — Budde told me that the team uses a combination of automation and direct human input. There’s a team running Full Coverage, which packages Yahoo content with links to outside news sources and resources on the Web. There’s a team for Sports, for Finance and for the content deal with SBC. There isn’t a 24-hour news team, though many editors have pagers that will go off in case of a breaking story.

“My [past] experience is with more people doing things and less automation, but the folks who have built Yahoo News up ’til this point have done a great job of being very smart about where they can automate and how much they can automate,” Budde said. “That frees up the human editors to do what human editors should do, which is make editorial judgments.”

Budde explained that Yahoo’s longtime partners such as the Associated Press and Reuters have helped streamline the automation process by working closely together with Yahoo News. “Going back many years, when online news was first developing, Yahoo News was one of the first to educate people like Reuters on what’s the best way to build their feeds for online products as automatically as possible,” he said. “So you can be smart about having the human editors do intelligent work on top of that.”

If you assume that the most trafficked Web pages are Yahoo.com, MSN.com and AOL’s proprietary home page, then you can deduce that news headlines on those home pages will generate the lion’s share of traffic to those sites’ news pages. Budde wouldn’t break down exact numbers on originating traffic for Yahoo News but said Yahoo.com, Yahoo search, and My Yahoo are key drivers.

According to Nielsen/NetRatings, Yahoo News is the second most trafficked News & Information site on the Web — and top Current Events/Global News site — with 20.8 million unique visitors in February 2005. Over the past two years, the site’s traffic has hit peaks and valleys depending on news stories such as U.S. elections and the Southeast Asian tsunami. But generally, traffic has gone from 16-19 million uniques in 2003 to the low 20 millions in 2004, while time spent on the site has gone from the high 20 minutes to low 30s.

One way to build on the time spent on the site is to make the story pages more rich with links to relevant content on other pages, according to Budde. He noted there’s a shift under way for news sites to treat story pages as entry ways for so many people entering via searches.

“People don’t start from the front page, they often start at story pages,” Budde said. “So if this is the first point of entry into our site, what can we do to expose people there to more of what we have available? I think it’s important for any news site. [We want to make] that the central focus, as opposed to so many editors who want to focus on the front page because that’s what the traditional editorial role has been — the front page.”

Taking RSS to the masses

While Really Simple Syndication (RSS) technology hasn’t really caught fire in the public consciousness, Yahoo has integrated RSS feeds into its personalized offering, My Yahoo, without really having to explain it. You just add content to your page by topic or by Editor’s Picks — of course, heavily weighted with Yahoo-hosted content.

That integration, started in January 2004, was just one of many announcements by so many online media outlets that have launched RSS feeds or a branded RSS reader. But this move by Yahoo had huge implications, because more than anything else related to RSS, this really did bring RSS to the masses.

“[Yahoo’s] integration of RSS into My Yahoo and Yahoo generally is so profound in terms of what it means for Yahoo,” said John Battelle, author of the upcoming book, “The Search” and founder of the Industry Standard magazine. “Yahoo News, in the Web 1.0 way, was supposed to be your RSS aggregator. Now, they still do Yahoo News, but they’ve given the reins over to users to do feeds that aren’t necessarily blessed by Yahoo.”

So now those feeds have spread from My Yahoo to a mobile service, a desktop ticker and to Yahoo News — probably the most appropriate place for them. Birkeland, the product manager for news at Yahoo, said that they have to perform a balancing act on giving people the content they want without giving them too much to handle.

“You could give people a lot of content, but we want to avoid what we call ‘firehosing’ people,” Birkeland said. “We don’t want to point the firehose at people and let them figure out what’s going on.” But he’s excited about giving people RSS feeds at Yahoo News, which “literally will allow your own choices to exist right in the middle of our front page.”

Still, there are limits. While you can put your feeds into news sections, you still can’t pre-empt the top story of the day. Birkeland says it’s important that Yahoo News still has enough control to convey important breaking news and allow serendipity — so people might learn things from unfamiliar sources and not just from an echo chamber.

While Yahoo News does have its own smallish newsroom in Building F, the operation draws on technology and resources from around the company. Scott Gatz, senior director of personalization products at Yahoo, has become known as “The RSS Guy,” bringing feeds to various Yahoo initiatives, including a search engine for RSS feeds, and the Yahoo News redesign. He told me the key to success for winning over the masses was not bothering to even call them “RSS feeds.”

“If you look at how we’ve integrated RSS into Yahoo News, we’re not actually using those three letters very much,” Gatz said. “So you look at it, and it says what would you like to add to your political news, here are some political blogs. Would you like to add CNN or MSNBC onto your news page? The fact that it happens in XML or RSS isn’t the important thing. Most of the users don’t want to have to figure that out.”

Skirting legal issues

Gatz says that Yahoo has more people using RSS than any other service, a number of users that’s “in the low millions.” He estimates there are 6 million RSS feeds around the Web. Bloggers take note: When people go to choose RSS feeds at My Yahoo or within Yahoo News, the Yahoo editors suggest various sources, from mainstream news sites to niche content to blogs. Getting into those slots likely will draw more traffic than from any other feed source.

Battelle, who also manages the popular BoingBoing blog, says that he has a contractual deal with Yahoo — with no money changing hands — where Yahoo lists BoingBoing as a feed to add, and BoingBoing has an “Add to My Yahoo” button on its blog. Those buttons are sprouting up like wildflowers on popular blogs now. But Yahoo allows its users to add any feed they want easily, and puts the RSS search engine right on those pages.

While Google was recently sued by Agence France Presse for listing its content in Google News without licensing it, Yahoo News has a deal with AFP and isn’t worried about causing trouble by adding RSS feeds to the mix. Why? Because any source that includes RSS feeds would want to have as many people add them as possible. Just having RSS feeds alone implies an invitation to run headlines on a news reader, Birkeland said.

“There is a clarifying effect of money, that Yahoo is willing to employ [with AFP and others],” Battelle said. “We’re going to syndicate your stuff, and therefore we’re going to pay you for it. And if you have a free RSS feed, we’re going to allow anyone to add it to it.”

While the BoingBoing deal doesn’t include money for placement, that deal is only short term. In the long run, Yahoo might be able to monetize where it places RSS feeds for users to add, just as it eventually charged money for e-mail services, personal ads and personal home pages. Just the fact that Yahoo editors are making a directory of RSS feeds and picking out the best blogs could eventually make it the de facto place for finding and sorting Weblog and niche content.

While a million RSS readers and startup companies have elbowed each other for attention, Yahoo has quietly become the elephant in the room for news feeds.

“We’re the first portal to do anything with RSS, and we’ve now had a year and a half to refine and improve it,” Gatz said. “The ultimate goal was to bring it to the masses, and now that we’ve learned that and made it easy, how do we extend it across the Yahoo network? The concept is powerful, that anyone can subscribe to anything.”

While Yahoo News has a long history of filtering and aggregating content on the Web, the future holds even more promise, with Moore and Budde on board, the resources of a forward-thinking technology company behind them and the new synergized headquarters in Santa Monica. The key will be their balancing act of human editing and automation, of collating content and helping millions understand what’s important in the news today.

* * *

The New Yahoo News
A rundown of some of the new features of the redesign, due for public beta later in April:

• “Toggle” feature lets you see news headlines from a particular news source at a glance.
• “My Sources” tab on each section lets you see your RSS headlines — and you can add feeds to the page.
• Tabbed navigation on top of each page, with more weight and space given to the top story package.
• Y!Q technology embedded into key words in stories brings pop-up box with links to relevant Yahoo and Web pages.
• Story pages will have one square ad within story instead of a top and side banner.
• “IM story” option lets you send a news story via Yahoo Messenger instant messaging service.