Comments on: Battling the 'bots? Try bringing out the big guns http://www.ojr.org/battling-the-bots-try-bringing-out-the-big-guns/?utm_source=rss&utm_medium=rss&utm_campaign=battling-the-bots-try-bringing-out-the-big-guns Focusing on the future of digital journalism Sun, 07 Apr 2013 15:02:13 +0000 hourly 1 http://wordpress.org/?v=3.5.1 By: Robert Niles http://www.ojr.org/battling-the-bots-try-bringing-out-the-big-guns/#comment-428 Robert Niles Wed, 14 Dec 2005 11:55:57 +0000 http://www.ojr.org/?p=947#comment-428 Metrics becomes a huge issue here. Webmasters routinely knock survey-based readership numbers, often with comment along the lines of “I know what my traffic is — just look at my log files!” But if you are not factoring bot traffic out of your log files, you are almost certainly overstating your human traffic, and possibly by an order of magnitude, if your site is getting hit like WebmasterWorld’s.

When I took over OJR, and spent some time with our log files, I was surprised to find that almost three-fouths of the site’s traffic was bots. I’d had sites get hit hard by bots before, but never as much as OJR was. Now, serving that traffic wasn’t a problem for us, and we’ve taken only a few, standard steps toward keeping out the bad guys. But it did require my doing quite a bit of work setting up new log file filters when we switched ISPs, in order to get more accurate traffic numbers.

]]> By: Barry Parr http://www.ojr.org/battling-the-bots-try-bringing-out-the-big-guns/#comment-427 Barry Parr Wed, 14 Dec 2005 09:46:19 +0000 http://www.ojr.org/?p=947#comment-427 Coastsider was nearly booted by my hosting service because of robot-related traffic. They were also wildly inflating my traffic numbers, making it difficult to set realistic ad rates. I was able to get my cpu load down to a reasonable level by caching some content, excluding bots from certain directories, and excluding certain bots altogether. It was a painful and time-consuming process and one I doubt is complete. But I could not afford to exclude robots altogether.

This also raises the question of how much robot traffic advertisers are paying for.

]]>