Please Feed The Spiders

Back in the day, if a website developer wanted to generate lots of traffic on their website, they could easily load their keywords, description, and main content with as much ‘popular’ content as they wanted such as “mp3”, “free software”, or “Brittany Spears.” The search engine spiders would then take all of this information home with them and store it in their database without so much as blinking. Someone using that search engine would search for “mp3”, “free software”, or “Brittany Spears”, and presto – that website would eventually show up as a relevant result. But if the website was really just trying to sell coffee online, what value was there to the person who came across their site? The person looking for MP3’s might end up buying coffee online, but probably not. In fact, that person might just feel (like millions of other avid search engine users), that they were deceived in some way and never visit or refer that site again.

Today, search engine spiders are much more intelligent than in years past. They now have the ability to cross-reference keywords with content, and in doing so ignore those sites which are trying to deceive surfers with irrelevant or popular content in an attempt to generate higher volumes of traffic. Spiders also look for things like hidden text (i.e.: trendy text with an identical color to the background color of the site), so the user can’t see it through a browser, but a spider can. These practices are now very much prohibited amongst most of the major engines.

Aside from just deceiving users, the search engine companies themselves are at risk by allowing such activities because in today’s online environment, there is increasing pressure for them to provide relevant results to their users.

However, there are some very accepted techniques for website developers to ensure the search engine spiders have lots of relevant content (A.K.A. “spider food”) to take home with them. One proven way is to ensure that all of your image ALT tags contain good relevant terms and phrases. Not just your main images, but all of them, such as any spacer graphics you happen to use. Keeping the ALT attributes in all your images with meaningful and relevant content is also a great way for you to increase the amount of content for your site if you don’t have a lot of content to begin with. Quite a few sites out there have something called a splash page – not a bad idea, except from the spider’s eyes, graphics alone don’t help, but loading the ALT’s with content will. Remember to feed the spiders, they’re appetites are insatiable.

Appropriate metatags are another way to keep the spiders interested. As I mentioned before, if keyword metatags don’t have anything to do with your main content, the spider may leave and not come back for a while. The best thing you can do when generating your keywords is to look at them objectively – NOT SUBJECTIVELY. Just because you think “columbia” is a great keyword for your online coffee store, does not necessarily mean people are searching for coffee using this word. Would you search for Columbia Coffee? I suggest by having the keyword “Colombian” in your main content you will be more relevant.

Getting the spiders to stick around is not hard, it requires good textual content (spider food), an up-to-date understanding of how the engines work, having all the proper HTML formatting in place (spiders need a road map too) and above all – an objective and patient perspective when designing your site.