Almost all Web publishers successful enough to have to pay bandwith charges have struggled with how to deal with traffic from robots. These are the automated programs, sent by search engines, crackers, spammers, sloppy developers and even overeager handheld owners, to scan, index and even download thousands of pages from your website.
When I arrived at OJR, I was surprised to find that more than half, almost two-thirds, of the site’s traffic was not from human readers, but from robots. Some of that traffic was welcomed, such as robots from major search engines like Google and Yahoo News. But much of it was from rogue spiders — spammers trolling for e-mail addresses, attempts to download the entire site for duplication on various scraper sites, and such. I spent a fair amount of time tweaking OJR’s robots.txt file to ban identified rogue spiders, and OJR’s stats software to filter hits from the rest.
Well, this week WebmasterWorld.com has taken the radical step of banning all spiders from its site. In a post on the site, administrator Brett Tabke reported that despite spending five to eight hours a week fending off rogue spiders, the site was still hit with 12 million unwanted spider page views last week.
The move, presumably, will result in WebmasterWorld disappearing from major search engine results and would eliminate the site from archive searches, as as Archive.org.
WebmasterWorld has established a large and loyal audience. One could argue that the site doesn’t need search engine traffic. But how loyal will its readership turn out to be if members can’t search for the site, or its archives, through Google, et al?
As Brett titled his post announcing the change, “lets try this for a month or three…”
Then we will see.








