Are full RSS feeds now more trouble than they are worth?

April 25, 2011 by

Are full RSS feeds now more trouble than they are worth?

I wondered that last week as the umpteenth Google alert hit my email in-box with a link to another “blog” that had scraped the full content of my posts. Curious this time, I clicked through and found something interesting at the bottom of the post.

It was the same list of social media links that I’ve opted into appending to the bottom of my posts in my Feedburner RSS feed. Their inclusion confirmed to me something I’d long suspected, but shoved to the back of my mind, that scrapers are using the convenient XML formatting of RSS feeds to populate their spam webpages.

(Let’s continue down the stream of consciousness, shall we?) That prompted me to wonder how many actual human beings are reading my site via RSS feeds today, versus spam bots harvesting those feeds to steal my content for their websites. With the rise of Twitter and Facebook, those have become the go-to sources for me to push new posts to my readers. Does anyone still use RSS?

It’s tough to answer that by looking at Feedburner stats. Perhaps an OJR reader with this information might inform us in the comments, but I don’t know of a good way to parse that data to separate human readers from scraper bots.

But the presence of so many scraper sites on the Internet, even after Google’s much-hyped Panda update, inspires me to consider cutting off their source of content. What if I killed my RSS feeds? Would the scrapers leave me alone? Would Google and Bing still find my content? Would my readership suffer?

Sitemaps provide a superior way to use XML to alert search engines and legitimate aggregators to new posts and content on a website, so I don’t believe the loss of an RSS feed would hurt you there. As I mentioned before, Facebook and Twitter provide new, more popular avenues for pushing new URLs to your readers and fans. But without an RSS feed breaking down your site’s content into easy-to-parse XML, scrapers likely would have a harder time extracting readable content from your website to put on theirs.

One interesting fact about the way that scrapers mine RSS feeds: They take only the headline and content, never the link. So as an interim step before killing off my RSS feeds, I’ve tried modifying them instead. I’ve rewritten the script that generates my feed to add the following line to each post in the feed:

“The article originally appeared at HYPERLINKED_URL_HERE. If you are not reading this post on a personal RSS reader (such as Feedburner) or on HYPERLINKED_WEBSITE_NAME, you are reading on a “scraper” site that has illegally copied our content. Please visit HYPERLINKED_URL_HERE for the original version, which includes all the reader comments.”

This places the original URL, and links it, within the copy of each post. Not only should that help search engines to know the canonical URL when the piece is scraped, it should help drive some of the scraper sites’ traffic back to my website. Ultimately, I don’t care about scraper sites if they drive their traffic back to me. It’s just when they take my content without returning traffic that offends me.

I just made this switch, and I’ll report back if I see any change in traffic, search engine placement or scraper abuse of my websites, as a result. In the meantime, I’d like to hear what you’re doing (or not) with your RSS feeds to fight scraping.

poll by twiigs.com

Comments welcomed!

Filed Under: Frontpage Tagged With:

So why aren't you Twittering yet?

January 9, 2009 by

Today’s entry is for all OJR readers who aren’t on Twitter yet. Here’s my advice: Start. Today.

Twitter has become what many of had hoped RSS would be, as well as the most vital forum for sharing links with other writers. Throw in Twitter’s value as the ideal medium for breaking news, and you’re crippling your online publishing effort by not participating.

Many journalists I’ve spoken with don’t “get” Twitter, due, I think, to its absurdly simple interface. You answer the question “What are you doing?” in 140 characters or less. That’s it. You cam post an update whenever you’d like and once an update is up, that’s it. No editing after the fact. (Which should be a familiar feeling to any print or broadcast journalist.)

You pick other Twitter users to follow, and see their updates on your Twitter home page. Your updates display on your profile page, as well as on the home pages of whatever Twitter users choose to follow you.

Unlike on Facebook, “friending” someone on Twitter does not have to be mutual. You can follow people without them having to follow you, and vice versa. However, smart decisions about whom to follow can help you greatly expand the network of individuals following you. I’ll write more on that in a few moments.

The next step beyond RSS
Let’s first look at Twitter’s ability as a medium to deliver your content to other readers. This is what RSS was supposed to do, and did reasonably well for the people who downloaded readers to access RSS. But Twitter is the application, not a protocol, so I’m finding many more “average” (i.e. non-tech-head) people using Twitter than ever read RSS.

Twitter’s 140-character limit also impedes publishers’ ability to simply scrape their story database for Twitter updates (“tweets”), the way you could to create an RSS feed. This might seem like a pain from a development perspective, but for usability, it’s a great thing. The short messages force publishers to write for the medium, making tweets far more to-the-point than RSS headlines, scraped from other media, have been. That rewards users with Twitter feeds that typically are much more compact and information-rich than RSS. I suspect that’s why average readers are embracing Twitter more than they did RSS.

And as handy as Twitter can be on the Web, on mobile devices such as iPhones, it is absolute information crack. I have downloaded the TwitterFon and Twitterrific applications on my iPhone (one for each of the feeds I maintain – twitter.com/robertniles for online media stuff and twitter.com/themepark for my theme park website) and check them with more enthusiasm than I check e-mail, Facebook or the Web. I’ve given up on my RSS reader, and haven’t checked it for months. (You can find OJR tweets on KDMC’s Twitter feed.)

One final argument for publishers: There is no “full-text” option in Twitter. Readers interested in your content click through to your website, allowing you to track and earn revenue from them far more effectively than most publishers could do with RSS. To see the power of Twitter as a story distribution medium, go to http://twitter.com/nytimes and start following.

Fishing with linkbait
If you have a large, established audience, you can throw a Twitter feed link on your navigation and quickly build a Twitter following. But the service’s real power lies in the way its social network can help you build an audience among people who are not currently reading your site.

This is where you need to be smart about whom you follow. Find other Twitter users publishing on your beat. (Twellow or Twitter’s search engine are good places to start.) Then click the button on their profile pages to start following them.

Since Twitter notifies you when someone starts following you, some percentage of those users will reciprocate, and start following you.

This is a great way to keep tabs on what others on your beat are doing, and writing. And, conversely, these other writers will now be seeing what you are doing, and writing. Your links become fodder for their blogs, and their links become fodder for yours.

Forget the hassle of e-mailing other webmasters, trying to secure valuable backlinks. That market’s now moved to Twitter.

About that reciprocity thing. For good Twitter karma, follow those people who follow you. Since I got the hang of this, the only folks I don’t reciprocate with a follow are

  • People without a personalized icon for their feed,
  • People with few or no updates in their feed,
  • Feeds that seem to me to be commercial spam, or with no content related to my beats.

    Consider that your “do not do” list, then.

    Breaking news
    I recently tweeted the Rose Parade on one of my Twitter feeds, and had a blast. Using nothing more than my iPhone, I used Twitterrific and its photo upload capability to post parade photos and text updates. By using the #roseparade tag in each post, I made them easy to find by readers using Twitter’s search engine. (Though, I must confess, I forgot to add the tag to my first few updates.)

    The use of the # sign before a word makes it a tag in Twitter, allowing readers to follow news threads from multiple writers. Granted, the Rose Parade is a lousy subject for breaking news online, as it happens on a holiday when almost everyone interested in it is at home with access to a TV to watch it live. But it was a great training exercise for my first attempt at live Twitter coverage. Armed with an iPhone with a Twitter app, I can cover any breaking news event I happen to be at, with text and photos, and post immediately to the Web. That’s how Twitterers provided many of initial reports about events such as the Mumbai attacks.

    Jump in. Try it. As my colleague Steve Buttry wrote, “Try it actively (10 tweets and 10 new follows/day) for a week and your view of communication will change forever.”

    I haven’t gotten to 10 tweets a day yet. But even with one or two a day, I’ve found Twitter delivering new readers, new in-bound links and fresh content for my websites. The only frustration I’ve had with it is the number of people, sources and agents I want to follow who are not yet using Twitter.

    So here’s my plea to you: Start!

    Update: (2009/1/16): This morning’s print LA Times features Janis Krums’ iPhone photo of the US Airways crash on the Hudson River in Manhattan, first posted to Twitter yesterday afternoon. I follow several LAT Twitter feeds and noticed the LAT make contact with Krums by replaying to the original tweet. Twitter is the multimedia breaking news wire now.

    • Filed Under: Frontpage Tagged With: ,

    Is Facebook the next frontier for online news?

    October 3, 2007 by

    Further empowering its users to grow its application ecosystem, Facebook recently announced the launch of the $10 million fbFund. Backed by outside investors, fbFund will grant $25,000 to $250,000 to selected individuals or start-ups building applications for the Facebook platform.

    A number of news organizations have already created Facebook applications to distribute their news content. The New York Times’ News Quiz application, which measures your daily news knowledge against your friends’, is installed on over 6,000 users’ pages and generates about 17,000 page views a week, according to NYTimes.com‘s Senior Vice President and General Manager Vivian Schiller.

    “This particular news quiz is part of a larger strategy to distribute content as widely as possible. There are different ways to engender loyalty and increase page views on the Web. It’s increasingly important to distribute content in parts and pieces, widgets and RSS feeds – wherever people want to consume it,” said Schiller.

    Indie start-ups are getting into the game, too. An app called “News Headlines,” authored by UK start-up RSS2Facebook, pulls in the RSS feeds of hundreds of global news providers and displays them in a single box. From there, news stories can be bookmarked or shared with friends.

    RSS2Facebook specializes in adapting the programming behind “News Headlines” for organizations which want to convert their existing RSS feeds into Facebook applications – a quick and dirty way to leverage Facebook’s immense social reach.

    OJR chatted with RSS2Facebook founder (and Southampton University student) Adam Cooke via MSN Messenger to learn more.

    OJR: How did you come up with the idea for RSS2Facebook, and when was it launched?

    Cooke: I had been creating some applications for small businesses, and a common request was to give individual RSS feeds a stand-alone application on Facebook. It didn’t have a launch date as such. I just found myself reusing the same code over and over for different people. So I decided to package it up with a helpful manual. The idea is that the content of your RSS feed is promoting your website. If you believe in your own content, then your application will spread quickly over the Facebook social graph – the mushroom cloud effect, if you will.

    So instead of having the RSS feed as a sort of output, it becomes a promotional tool. It spreads the word to the users’ friends. As friends are normally interested in the same things this can be seen as a form of extremely cheap, targeted advertising.

    OJR: How fast is “News Headlines” growing?

    Cooke: Over 1,500 users have the app installed. I have actually had no need to promote it. It’s amazing really. There is no opportunity for big business advertising on Facebook unless they are actually offering a service to the user. This makes it a very level playing field for small business vs. big business, because for the first time it’s the content that decides, not the promotional budget.

    OJR: What kind of interested have you gotten in RSS2Facebook’s services?

    Cooke: I’ve gotten loads. Small sites and news feed types. I have done some custom application builds for larger sites, like the online job agency for students, Graduate Prospects. It lets you watch job feeds from the agency in your profile. Very useful if you’re a student just coming out of university.

    OJR: Have you made any money yet?

    Cooke: In the four figures – put it that way. Hopefully five very soon.

    OJR: How did you hear about the fbFund?

    Cooke: I actually heard it through a developer IRC channel but didn’t get any details until I watched a piece about it on the local news. Well, it looks really good. Free money I guess. LOL. I would definitely apply.

    OJR: If you got a grant to work on RSS2Facebook full time, where would you take it?

    Cooke: I am currently working on an RSS2Facebook multifeed. I have some other ideas.

    OJR: What are some of the other ideas?

    Cooke: Well, if I told you, then I would be giving away my advantage, wouldn’t I? They are along the same lines as what I have now on RSS2Facebook. Some ideas involve the integration of well-known open source software with Facebook. It’s a very competitive industry. If this were a regulated industry, you would not be getting away with some of the copycat applications out on Facebook right now.

    But to be honest, I like it how it is. It’s open, and the best application wins – which is ultimately what Facebook wants.

    Filed Under: Uncategorized Tagged With: , ,
    Next Page»