Search is rapidly turning from the darling of the internet to a demon. The latest attack, brought about by search’s phenomenal financial success, is that search is sucking all the value out of millions of websites by scraping content in bite sized pieces and doling it out to searchers, at the same time monetizing traffic where the value is being created by the site owners, not the search engine.

Usability guru Jakob Nielsen ran a post on his site a few weeks back (http://www.useit.com/alertbox/search_engines.html) taking the engines to task for fostering dependency on search traffic, while at the same time enjoying the impact of ever escalating bid prices. A few analysts have also started to intimate that perhaps it’s time for marketers, especially those in direct response, to look beyond search as click prices move the cost per acquisition beyond what’s reasonable.

Usability: Too Much of a Good Thing?

It a somewhat ironic twist, Nielsen illustrates how improved usability can be a big factor in driving up bid prices. When a new online search market emerges, bid prices usually settle out based on the relative conversion performance of the main bid contenders. Based on their current conversion and closing rates, the smarter search marketers determine what they can afford to pay per lead. But let’s say that one of the contenders suddenly makes their site a better conversion vehicle, doubling their capture and close rate. Suddenly, that marketer can bid twice as much and still end up with the same per unit ROI. They become more aggressive in bidding, and eventually, their competitors wake up, figure out what happened and launch their own improvement cycles. Bid prices escalate as marketers optimize every aspect of their campaign, and the search engines double their revenue by doing nothing. In this scenario, not only are engines leeches on the content of the internet, they’re also sucking the blood out of the entire search marketing industry. The better we get, the more money they make.

Now, before we round up the lynch mob for the engines, there are a few things that have to be said on their behalf. In cases like what was described above, search acts as the initial match maker between a site and a prospect. True, if it’s a one time transaction, search can skim off a substantial portion of the value realized from sale, but if it’s a situation where lifetime value is a multiple of the first sale, the returns from that search lead will far outweigh the one time cost. Nielsen also acknowledges that traffic generated from organic referrals doesn’t add anything to the search engine coffers. While he dismisses this traffic source in one brief mention, I think it should be realized that search engines have continued to provide this service, devoting a substantial portion of their page real estate to organic listings. 3 out of every 4 users are still clicking on these listings.

Search: Providing Quick Answers

Another concern expressed by Nielsen, which has also been vocalized, very loudly, by the World Association of Newspapers (http://news.ft.com/cms/s/d0e8cf3e-928d-11da-977b-0000779e2340.html) is that many users use search as an answer engine. They need a quick answer to a question (i.e. What was Brad Pitt’s first starring role?) and often they don’t have to go any further than the search results page to find it. The answer is often contained in the snippet of text that comes from the listed site. If you did the search above, you’d find that the site tiscali.co.uk has the answer (Dark Side of the Sun, by the way). The content came from that site, answered your question and you’re merrily on the way, arguing about who played the corpse in the Big Chill. But that’s far from the experience the owners of tiscali.co.uk wanted. They pay for their site by selling advertising. On the page that snippet came from are three separate ads. You didn’t see any of them, but you did see the ads the engine chose to show you.

Is this a grey area? Absolutely. But the organizations that are investing in the production of that content that the engines so nonchalantly pilfer are getting tired on contributing to Google’s ever increasing bank accounts. While publishers (including book publishers) acknowledge that the search engines are necessary for connecting users with their content, they don’t want those same users stopping at the search results page, having found what they were looking for. Gavin O’Reilly, president of the World Association of Newspapers, summed it up, “We need search engines, and they do help consumers navigate an increasingly complicated medium, but they’re building (their business) on the back of kleptomania.”

The New Golden Rule of Online

While I understand the frustration, I think it’s time for a reality check. The fact is, it’s the new User Rule for online: Those that have the users, make the rules.

Search works. Users know that. With that come some positives, and some negatives. The way I see it, search brings much more to these content sites than it takes away, so it’s not a parasitic relationship. Would these sites be happier if the search engines didn’t index their content at all? If so, that’s doable with a simple robots.txt file. If not, quit your whining and get used to it!