Several times a week, I open my snail mail box to find bulk-mail solicitations for some member of one of my websites, but sent to the site’s street address. Every month or so, I’ll get a series of calls to my business phone (which is listed on my website), but the caller will ask for a name I’ve never heard. For the rest of that week, I’ll get dozens of similar calls, from different people calling on behalf of some work-at-home scheme, all asking for the same fake name.

And whenever I’m stuck searching for information via Google or Bing, I inevitably have to scroll past link after link to scraped websites – pages written not by any human being, but slapped together by scripts created to blend snippets from other webpages into something that will fool Google’s or Bing’s algorithm into promoting them.

If Google really wants to make its search engine results pages more meaningful, forget about adding links from my Google+ friends. How about creating a scraper-free search engine, instead?

I have no doubt that the reason why I get all those misaddressed letters and wrong-number phone calls is that some fly-by-night “data” company scraped together a database by mashing up names, street addresses and phone numbers it crawled on various websites. That database gets laundered through some work-at-home company, which sells it to customers suckers via the Internet as a “lead list” for commission sales.

It’s bad enough to take phone calls from these poor chumps, who think that they’ve taken a step toward earning some honest income. But I’m stunned when I see the bogus-name letters coming to my office from established colleges and non-profit institutions, who clearly also have bought crap mailing lists.

(FWIW, all my phone numbers are on the National Do-Not-Call Registry, and I’m opted out of commercial snail mail with the Direct Marketing Association, so no legitimate data company should be selling my contact information to businesses and organizations I’ve not dealt with before.)

Maybe it’s too much to hope for a solution that frees me from having to throw away all these unwanted letters and beg off these unwanted phone calls. (Not to mention saving the people contacting the expense of pursuing bogus leads.) But maybe I can hope for a scraper-free Internet experience instead.

I know it’s possible, because there used to be a scraper-free search engine – one that searched just hand-picked Web sites created by actual human beings. It was called Yahoo!, and if they’re smart, the latest crew of new managers at Yahoo! could do far worse than trying to recreate a 2012 version of their Web directory, then using it to populate a Google-killing search engine.

For an example of the garbage polluting search engines today, this site came up high in the SERPs when I searched recently for my wife’s name and the name of her website.

If you know anything about the violin, you should be ROTFL now. For those who aren’t violin fans, allow me to explain that Ivan Galamian, one of the great violin pedagogues of the 20th century, has been dead for over 30 years. While we would have loved to have someone of his stature working for us at Violinist.com, only an idiot scraper script would think he works for us now.

It kills me that good websites, blogs and journals written by thoughtful correspondents get pushed down in the SERPs – and overlooked by potential fans – because of this garbage.

I want a search engine that knows better – that excludes Web domains populated by scraped data and instead searches online sites written by actual human beings. I wouldn’t limit such a search engine to sites written by paid, professional staff. There’s too much rich content to be found in the conversations of others. But blogs, discussion boards and rating-and-review sites included in this search engine should be composed of information submitted by human beings, not scraped from other websites and edited together by bots.

The original Yahoo! lost when start-up rival Google indexed more pages than Yahoo, giving Google an edge over its established competition. But I – and, I suspect, many others – don’t care about the size of a search engine’s database any longer. Google’s right on in its attempt, announced today, to build a more human-driven search engine. But I’m not convinced that adding Google+ links to the SERPs is enough of a change to make a difference in quality.

First, not enough people use Google+. Its 18-and-over-only age limit also disqualifies the millions of teen-agers who help drive the digital conversation. And I fear that Google’s new “Search Plus Your World” approach simply will encourage spammers to flood Google+ with even more bogus accounts and friend requests, in order to boost their reach into the Google SERPs those new “friends” see.

It’s great to use social media to help bring more people into the process of selecting which websites should be indexed in a search engine. But, ultimately, at this point organizations still need more aggressive in-house human oversight in back-checking the results.

Google lost its quality control over its SERPs long ago. Whether it’s search engine results or business lead lists, there’s too much crap data on the market today. That illustrates the continued need for more, and better, human leadership of data cultivation. There’s a market need out there. So who’s going to step forward to fulfill it?

Let’s talk about it. What would the deal mean for the online news business? For online entrepreneurs? For the economy?

[Sorry -- It looks like Twiigs.com, the company that hosts the poll, has eaten the results a couple times due to some server issues it's had over the weekend. So please do vote again if you see the input form below (which means that your old vote was among those eaten.]

For a magazine/Web outlet designed to bring information to readers, Wired sure selected a media-light crowd of exhibitors this year. Just eight out of 162 exhibits had anything to do with communications. And really, only Yahoo’s presentation had much of interest to anyone working in online media. (The rest were cool 3D displays, cellphone activated lightshows, installation art of instant messaging, etc.)

What gives? Where were all the next-gen social media applications, the iPhonery, the streaming video delivery stuff? NextFest opted for the wow-factor of robots and lightshows and missed out on what actually changes our lives.

I had a chat with Ben Clemens, Director of the Design Innovation Team at Yahoo, who also did a stint at the online portion of the New York Times.

Ben explained that his team is working on a unique app that will visually chart Web searches in real time and map them onto a model of the globe. Playing back the data will give an insight into how searches spread and develop over geographic space and over time. I thought it would be tremendously useful for journalists following the news cycle of a story, so I asked him about the model. (Partial transcript follows the video.)

Ben Clemens: The idea is there there search burst events which are lots and lots of people looking for the same thing at the same time and we want be be able to visualize that and show what’s the geographic pattern that they are looking for.

OJR: What sort of application might this have for tracking the way people follow a news story, for example?

Right now what you’re seeing is a fairly coarse level of data, but what we’d like to get to is the point where we can actually see as a story unfolded pegging the spread of search queries in some sort of more local event. One of the data sets that we’re actually working on right now is the bridge collapse, we wanted to track on a very local basis how it was that the searches spread, because that started as a very local event and then became a national event. Right now we don’t have the fine grain of geo-coding we would need to actually do that, but that’s the next thing we are working on.

OJR: Would then news websites want to tailor their news offerings based upon real time what people are interested in specific locations?

Ben: I think probably journalists will make their own decisions, but I think it’s good information to get from actual user data. This is what people are actually doing, as opposed to what they say they are interested in.

OJR: Does this connect with Yahoo News at the moment?

Ben: This is an experiment; it is not part of any Yahoo product. We would like to take advantage of it in Yahoo products going forward but for now we’re just at the bleeding edge trying to figure out how we would use this. Just the mechanism to get the data and to individualize it are a lot of the mechanics that we are working on right now. If that gets to a good enough state, then we would talk to products.

OJR: How far down the line is that?

Ben: (Laughs.) I really can’t say.

Ben then showed me an austere white-on-white globe of the earth with slow-moving blue specks shooting out from the surface of the North American continent. He explained that each speck represents a search query instance and that the speed and thickness of the particle streams indicate the popularity of the search. The data set at hand was a Yahoo search for “Mattel,” immediately following the lead-in-toys story that drove worried parents by the thousands to the Internet to search for their child’s toy.

Interesting stuff, and sure to give us too much information about ourselves down the line.

Other than than, NextFest was a bit of a bust from a journalist’s perspective. I mean, don’t get me wrong: the Google Lunar X Prize announcement was uber-cool (journalist was third on my list of childhood aspirations, astronaut and paleontologist being numbers one and two), but really, the lack of media eyecandy was disappointing. I would have thought it would have been a perfect fit for OJR–Wired is journalism that brings you technology and OJR is journalism about technology that brings you journalism–but eh, so it goes. I suppose I shouldn’t be surprised.

The ads are deemed relevant by the computer algorithm that might search the story for key words and match them to the key words that these advertisers bought. But the algorithm can’t quite match human judgment that might consider this to be the wrong place for legitimate spas to promote their services.

When Google launched self-service AdSense advertisements in June 2003, these types of mismatched ads proliferated — almost to the point of being a parlor game where you find the weirdest mix of ads and content. But over time, Google AdSense and Yahoo’s competing ContentMatch have improved their algorithms, have worked more closely with big publishers, and have taken feedback from readers and advertisers to improve their systems and reduce mismatches.

Even so, contextual advertisements remain a strange breed, living alongside journalistic content or blog postings as text ads and fighting for attention with rich media ads. Contextual ads have performed well for Google and Yahoo, brought in some good money for online publishers, and given the advertisers some bang for their buck. If all three parties are happy, that’s nice for their business, but how much tolerance should there be for goofy pairings?

John Battelle, who is starting FM Publishing to help bloggers sell ads on their sites, sees advertising as a conversation and believes that there’s only so much an algorithm can do in matching advertisers with publishers.

“I think it rings a sour note in the conversation in the publishing community [when mismatches happen],” Battelle told me. “And sour notes over time can lessen the conversation. I know that the platform guys [at Yahoo and Google] are working really hard to stop that problem, which is really difficult to do, because it’s language. … You can do a hierarchy and say if this is a news story, you show only these types of ads, but then you’re missing out on what might be the right ad.”

As Google improves AdSense, Yahoo is readying a similar self-service platform, which is only in limited beta currently. But Yahoo’s Search Marketing division (formerly Overture) has worked with large publishers for some time and has a long history of matching ads with search key words. Here’s a rundown on the good, bad and ugly of these contextual ad services — and which content works best in matching ads and bringing in more revenues for sites.

The good

The first rule of thumb for contextual ads is the opposite of the old local TV news maxim, “If it bleeds, it leads.” In the case of hard news related to war, terrorism, rapes, murders and other unpleasant current events, the best bet could be to remove contextual ads or just run more generic run-of-site ads. Of course, some site publishers have no problem with hawking commemorative Iraqi War playing cards on stories about people dying in the war.

Yahoo’s ContentMatch uses various sensitivity filters for what ads are shown with content — along with human oversight from a staff of more than 100 editorial people. Paul Volen, who is vice president of product marketing for Yahoo Search Marketing, told me that Yahoo’s human oversight is what differentiates his service from Google’s.

“First, we do have a dynamic proprietary algorithm which takes the theme of a page and matches advertising to it,” Volen told me. “We also add a layer of editorial oversight to that. We look at a site and try to understand if there’s content that’s evergreen and won’t change, like index pages that are generic and aren’t updated every day. In those cases, we could have the editorial staff look at them and come up with the best matches. We work with the publishers pretty closely across their site to figure out how we want to attack the relevancy and matching.”

Google doesn’t have an editorial staff for AdSense, but does assign account managers to bigger sites for the premium level of AdSense services — though it is only available for sites above 20 million page views per month. Gokul Rajaram, group product manager for Google AdSense, told me that there are just too many dynamic pages served for humans to watch every ad that’s served up.

So what content works well with contextual ad services? Usually the more focused the content, and the more commercial it is, the better. So a blog that covers Asian travel should outperform a blog about inner-city crime. Rajaram says there are a number of factors that affect how well the ads do for publishers and advertisers — including how fertile a topic area is for advertisers and where the ads are placed on a page.

“If the advertiser makes $100 per sale, then they’re willing to pay $5 per click,” he said. “If the advertiser only makes $10 then they would only be able to pay one-tenth of that. The cost-per-click depends on the vertical that the advertiser is in, as well as the competitiveness of the term that it’s in. … Finally, there’s the matter of clickthrough rates, which depends on the relevance of the ads and the placement of the ads on the page. What is the mode of the users, are they in a browse mode and just reading news articles?”

For a general news site, the breaking news sections might not perform as well as lifestyle, travel, business and technology. One great example of a fertile ground for contextual ads is Weblogs Inc., the network of blogs set up by entrepreneur Jason Calacanis. Calacanis has used AdSense to help jumpstart his business as he also sells display ads on his 80-plus blogs that are very targeted into vertical categories — from digital photography to babies and pregnancy.

For that reason, Calacanis has nothing negative to say about his experience with AdSense.

“The ads are absurdly targeted in my experience,” Calacanis told me via e-mail. “People look at them like content and they don’t even mind them. It’s a revolution in advertising –for the first time since the Super Bowl people want the ads! It’s advertising nirvana for everyone involved: advertiser, publisher, and Google. … Text-based advertisers find out about our site because of Google AdSense and then they wind up buying display ads. So, Google is doing free marketing for us in a way. I’ve found no downside as a publisher — zero.”

Still, Calacanis isn’t putting all his eggs in the contextual ad basket and realizes the majority of his income is from display ads sold directly to advertisers. Plus, he says Weblogs Inc. is considering using the FM Publishing network being set up by Battelle to help match advertisers with blogs using the human touch.

The bad

While automated contextual ad services take the hassle out of selling advertisements for small publishers or bloggers, they might not perform that well for medium-sized sites that aren’t focused on very specific commercial niches or cover more strategic or philosophical subject mattter.

Matt McAlister is vice president and general manager of tech trade magazine InfoWorld’s online operations and also runs the remains of the Industry Standard’s Web site. While the latter site does get about 50,000 unique users per month, according to McAlister, the AdSense ads just don’t seem to fit well or bring in much income.

“In terms of revenues, there’s certainly a threshold that your site needs to reach before [AdSense] does anything for you,” McAlister said. “TheStandard.com has 50,000 uniques, but [the income] is almost laughable. It’s better to do sponsorship, where someone owns a section of the site. I remember examples of the stuff that came up that was either completely abstract or inappropriate. I don’t think it has great value to the people who are using the site. With the site at 50,000 uniques, that should be attractive at a certain level, but I have a feeling it’s not a useful way to use your ad inventory.”

InfoWorld itself is owned by media conglomerate IDG, and its site uses the contextual ad services of a company called Industry Brains. When you click on an InfoWorld story, you’ll see small text ads placed adjacent to the first paragraph or two of the story. They look like AdSense ads but, they’re actually served and created by InfoWorld itself. McAlister says they decided to run these ads in order to help advertisers who wanted to generate leads. In many cases, he said, people tune out graphical ads or don’t ever click on them — but they’re more likely to click on text ads that are relatively relevant.

“We started off by targeting them at a very granular level,” McAlister said. “Our taxonomy has about 200-some terms in it, so we were targeting two or three levels down the tree. But we found that it was better to back out and only target the top level, the top 14 categories, because you’re able to reach a little bit further on the number of impressions you’re able to serve. The targeting at that level wasn’t that effective. They might not be as relevant, but with InfoWorld you don’t have to be that specific to convert ads.”

But McAlister does see the value in contextual ads and clicked on various ads while planning a recent vacation using Yahoo Travel.

“In a vertical like that, where the inventory is limited to a number of partners, then that scarcity creates value,” he said. “But if you’re using a blanket key-words matching system against the text on a page against an infinite number of sites, I don’t think that’s going to have much value for the publisher or the readers.”

Battelle, who also founded the Industry Standard magazine, agrees with McAlister when it comes to automated contextual ads. He reiterates the point that medium-sized publishers are often stuck in the cold with Google and Yahoo. And his experience with AdSense on the popular group blog BoingBoing was that the ads just didn’t fit with the often esoteric subject matter.

“Big publishers are slowly working out human relationships with the Googles and Yahoos of the world and start to filter and create feeds [of ads] that work for them,” he said. “But you can’t have a guy who has 250,000 readers a month as opposed to 2.5 million who can have a conversation with the AdSenses of the world.”

The problem is that a Web page with content on it — whether a reported article or a blog rant — isn’t the same as a page that has basic search results. So the interaction between key-word triggered ads and the content is different and more subtle to work out. That’s how Battelle sees FM Publishing squeezing in to serve high-traffic blogs.

“Search has proven that point with this great aggregator on one end of the conversation, where people were putting their question into a search box, and a magic list came up that appeared to be relevant,” Battelle said. “But what about the places where the conversation is already happening? A blog is not where someone is putting a question into a search box, but where a community is already very real, and you need to know the mores of that community before you can enter that conversation in way where you’ll be accepted.”

Jennifer Slegg runs a blog covering contextual ads called JenSense. Slegg compared ad relevancy using early versions of Yahoo’s self-service network and says they still have a ways to go to match up with Google AdSense. She also mentioned one other fault with the AdSense matching system — dynamic content pages aren’t checked by Google very often.

“Bloggers have a harder time with relevancy, particularly on their blog index page when they post frequently,” Slegg told me via e-mail. “Because Google indexes about once per month, writing a single entry about popcorn right before the [Google] bot visits can result in popcorn ads for about a month, even if the majority of the posts are about a tech subject. Targeting on individual blog entry pages is usually very good. This is why many of the news sites running contextual ad programs do not often run ads on the pages that change frequently, such as the index page and the subsection pages, but run it on the articles themselves.”

The ugly

While many of the early kinks of contextual mismatches have been worked out at big news sites, there still remains a gray area that’s difficult to catch. Just spend some time digging through the local news online, and you’re bound to come up with a doozy or two.

For instance, a story on LATimes.com about police spying on activists ends with ads for doing public records searches — basically spying on people. And below an obituary for a rabbi in NYTimes.com is an ad link to humorous Yiddish garb. While publishers and the technology companies might be watching for problems, they still slip through.

I talked with one of the advertisers who was included below the Mercury News story about the massage parlor busts. Pierce Salguero is executive director of Tao Mountain, a non-profit which offers courses in Thai massage and herbal medicine. He wasn’t sure why his ad came up in that case and told me that he actually used the terms “sex” and “prostitution” as negative key words in AdSense so the ad wouldn’t come up in association with these search terms or content.

“[Mismatches] probably happen more often than I know about which is unfortunate, because we’re not a spa,” Salguero said. “We’re a non-profit network of researchers who are working at the medical application and history and context of traditional medicine in Thailand.”

But despite the problems, Salguero wouldn’t give up using Google and Yahoo for lead generation for his programs. He says he gets 80 to 85 percent of his business through paid search and contextual ads online vs. print ads and the occasional direct mailer.

“It’s definitely been worth doing no matter what snafus might have happened along the way,” Salguero said.

One sign of the effectiveness of these types of ads is that the publishers themselves have been experimenting with buying paid search and contextual ads to promote their own stories. NYTimes.com and Washingtonpost.com have both been pretty aggressive at using key words related to stories they want to promote.

Tim Ruder, vice president of marketing at WashingtonPost.Newsweek Interactive, told me they have promoted stories as well as sections of the site. The only problem is that as this method of promotion becomes more popular, the cost of key words will go up and it will lose its allure.

“We’ve seen that happen across the board within search,” Ruder told me. “As it becomes more competitive, you make all this stuff based on return-on-investment decisions. The ability to make a healthy ROI is harder and harder as it becomes more competitive.”

One ugly side effect of this promotion is when competitors end up promoting their news stories as a place to get more information on a rival site’s ad box. While Ruder hasn’t seen any NYTimes.com promotions come up on Washingtonpost.com, he did briefly see a Washingtonpost.com promo on a NYTimes.com page. But these are the type of glitches that are easily reported to Google or Yahoo, or fixed through filtering — as long as you can keep up with all the pages served.

Ruder notes that Washingtonpost.com does already offer self-service ads for job listings and other classifieds but thinks the key word auctions at Google and Yahoo are a bit more complex to emulate. In the end, publishers of all sizes will have to weigh out how much return they’ll get from automated contextual ads and whether it makes sense to split ad revenues with the tech companies.