“Be careful what you wish for, it may come true”

While at Ad-Tech, I lamented the clogging of Google’s results with spam filled sites. I asked Google to clean up its index. Although I’m sure there’s no connection between the two (unless Sergei and Larry are paying a lot more attention to me than I thought) Google responded just a few weeks later with the Florida update. And boy, have they responded big time!

If you haven’t ventured into an SEO forum for awhile, you might not have heard of the Florida update. It’s Google’s latest dance, and it’s a doozy. It appears that Google is trying to single handedly shut down the entire affiliate industry.

The scene is awash with guessing and speculation. Was it a Google mistake? A plot to force advertisers to move to AdWords for placement? Barry Lloyd did a good job of trying to bring sense to the mayhem. I’d like to jump in with some further research we’ve done and my own thoughts of what’s happening with the Google index.

A Florida Guide

First of all, the Florida update was rolled out November 16th. It appears to be a new filter that is applied to commercially based searches, triggered by certain words in the query. The filter clears out many of the sites that previously populated the top 100. In several tests, we found the filter generally removes 50 to 98% of the previously listed sites, with the average seeming to be 72%. Yes...that’s right: 72% of the sites that used to be in Google are nowhere to be seen!

Who’s Missing?

The target is pretty clear. Its affiliate sites, with domains that contain the keywords, and with a network of keyword links pointing back to the home page of the site. The filter is remarkably effective in removing the affiliate clutter. Unfortunately, legitimate commercial sites with lower page rank are being removed as well. There seems to be a PageRank threshold above which sites are no longer affected by the filter. We’ve seen most sites with PageRank 6 or above go through unscathed.

And the Secret Word is…

The filter also appears to be activated only when search queries contain certain words. For example, a search for “Calgary Web Design Firms” activated the filter and cleared out 84% of the sites, while a search for “Calgary Database Development” didn’t activate it. Search volumes are roughly equivalent for both phrases. The filter seems to be activated by a database of phrase matches, and doesn’t appear to be affected by stemming. For example, “Panasonic fax machines” activates the filter, but none of these words as a single search phrase does. “Fax machines” activates the filter, but “Panasonic machines” doesn’t.

Also, it seems that only a few single word searches activate the filter. We found that jewelry, watches, clothing, swimwear, shelving, loans and apartments all activated the filter. Other terms that you would think would be bigger targets for spam, including sex, cash, porn, genealogy, MP3, gambling and casino don’t activate the filter. Obviously, when you look at these words, Google is more concerned with commercialization than spam.

Volume, Volume, Volume

Another factor is whether the filter is tripped or not seems to be search volume. Any commercial searches with volumes over 200 per month (as determined by Overture’s search term suggestion tool) seemed to trip the filter. Searches under that threshold seemed to remain unfiltered. For example, a search for “Oregon whitewater rafting” (about 215 searches last month) activated the filter, while a search for “Washington whitewater rafting (about 37 searches last month) didn’t.

What is Google Thinking?

Obviously, given the deliberate nature of the implementation, this isn’t a hiccup or a mistake by Google. This was a well thought out addition to the algorithm. And in the most competitive searches, it produces much better results than did the “pre-Florida” index. If you search for “New York Hotels” today, you’ll find almost all of the affiliate clutter gone.

Where the problem occurs is in the less competitive searches, where there’s not a sufficient number of PageRank 6 or higher sites to fill the vacuum caused by the filter. If you do a search now for most phrases you’ll find the results are made up of mainly directory and irrelevant information sites. In cleaning house, Google has swept away many sites that should have stuck. As an example, visit Scroogle.org and search for “Calgary web design firms”. Scroogle is from the deliciously twisted minds of Google Watch, and gives graphic representation of the bloodshed resulting from Florida. In the pre-Florida results, the top 10 (all of which were wiped out by the filter) included 6 Calgary based web designers and 1 in Vancouver (two of the remaining results were additional pages from these firms). The other result was a directory called postcards-usa.com with a page of design firms from around North America. Eight of the 10 results were directly relevant, one was somewhat relevant and one was of questionable relevancy for the geographically specific search.

In the filtered results, there is not one web design firm from Calgary. The top 4 listings are directory site pages, two of which are not even specific to Calgary. Ranking 5 and 6 belong to Amazon.com pages selling a book on web design (nothing to do with Calgary other than a reader review from someone who lives there). Rankings 7 and 8 go to pages about evolt.org, a non profit organization of web designers, and a profile on a Calgary based member. Listing 9 goes to the web design page of an abysmal web directory, again not specific to any region. And listing 10 goes to an obvious link farm. Of the 10 results, none of them were relevant.

Google’s Next Move?

Pulling out the crystal ball, which in hindsight was amazingly accurate 2 weeks ago, here’s what I think will happen. The Florida filter will not be revoked, but it will be tweaked. It’s doing an amazing job on the ultra competitive searches, but the algorithm will be loosened to allow inflow of previously filtered sites to bring relevancy back to the less competitive searches. Hopefully, the sites finding their way back into the index will be better quality legitimate commercial sites and not affiliate knock offs. Google has to move quickly to fix the relevancy for these searches, because they can’t afford another blow to the quality of their search results.

I really don’t believe that Google purposely implemented the filter to drive advertisers to AdWords, but that is certainly a likely side effect. The most dramatic impact will be the devastation of the affiliate industry. Just 3 short weeks ago I listened to 4 major internet marketers say they didn’t bother with organic SEO because their affiliate partners did it for them. Those days are over. If Google was targeting anyone with Florida, it was affiliate sites. A number of forum posts indicated that Google was taking aim at SEO. I don’t believe so. I think Google is trying to wipe out bad SEO and affiliate programs and unfortunately there are a number of innocent bystanders who got hit in the crossfire. But every indication from Google itself (both from posts to forums and in replies to help requests) seems to indicate that Florida is a work in progress.