Administrator's Corner

by Richard Zwicky

All SEO Articles

This article from The Mender (Issue 15),
Metamend's Web Site Optimization and Marketing Newsletter.


Last month, for the first time, I did not make a contribution to the newsletter, for a variety of reasons. This month, I think it's my longest contribution. I hope you find it relevant, topical and interesting. It's commonly accepted in the search engine optimization (SEO) industry, and by any avid web site operator who spends a lot of time researching the subject, that getting listed in major search vehicles, such as Altavista, Google, GOTO, Looksmart, Lycos, Hotbot and others, is crucial to the success of a web site. It's also obvious that it's not as straightforward as entering your URL, and hitting the "submit my site" button.

But what most people and resources don't mention is that there are a variety of strategies, and many other less known search engines coming on the scene which will give you a lot more exposure than you might expect. Not just new search engines, but also lesser known ones and new strategies on the scene such as the pay for inclusion models being used by some engines, like Inktomi, Looksmart, and Altavista, and of course the slightly older pay for placement model such as employed by GOTO.

What are the different models, and how to succeed at them? Well here's a little sampler of the lesser known engines and also of the models employed by some of the majors. Oddly enough, and happily for us, success is almost always predicated on a proper web site optimization strategy designed to ensure that relevant content is presented to the search engines properly, in the correct formats and without contravening any of the rules.

The lesser known engines include DMOZ, and AlltheWeb (FAST). What makes these unique and worthwhile resources? To start with, very few people ever use DMOZ - also known as the Open Directory Project - to run a search, but DMOZ runs their business by selling their data to many other major and minor search engines, so their reach is extensive and powerful. For example, the last time I checked, DMOZ data was known to find its way directly into the search indexes at Altavista, AOL search, Ask Jeeves, Direct Hit, Google, Hotbot, and Lycos.

Now take the data one step further: Altavista supplies data to Ask Jeeves; Ask Jeeves supplies Altavista and Direct Hit; Direct Hit supplies Ask Jeeves (!) Hotbot, Lycos and MSN; Google of course supplies Yahoo!, and Netscape search, (interesting how AOL and Microsoft use the same resource indirectly). I could keep on going, but the point should already be obvious. A proper submission in the
right places can go a long way. Submitting a site which is not properly optimized will possibly get indexed, but probably won't help you much. Properly optimizing your site, and keeping it optimized for the search engines will make all the difference.

My other example, FAST (AlltheWeb) claims to have indexed over 600 million web pages and to have spidered and crawled over 1.6 billion URLs. It also receives over 7 million queries daily, meaning it probably is well known and used, even if it is rarely mentioned in most discussions. There is one claim that strongly differentiates it from other search engines; it refreshes its search index every 9 to 12 days, which is much more frequently than other search engines claim. Their submission process is easy. Go to their submit page, fill in the URL, and your email address. Then wait.

The time it takes to be indexed will vary, depending on the number of pages in your site, whether the site has already been visited, where is sits in the submissions queue, and of course, server load. Occasionally, first time submissions are indexed within a two to three weeks. If your page is improperly formatted for FAST's rules, there is no guarantee that the site your submitted will be indexed, or properly indexed. This will change in the second half of the year as FAST move to a pay for inclusion model. Your site will be guaranteed to be spidered, but that you show up anywhere will still depend on proper formatting.

This pay for inclusion model, has been adopted in it's cleanest form by some better known resources, such as Inktomi. Inktomi's search engine service, which no one visits to perform a search, has an incredible reach. Inktomi really is a pioneer in the concept of a pay for inclusion model which demands proper formatting and presentation of data. Their model guarantees your site will be visited on a regular schedule which may be as short as every 48 hours. If your web site is properly formatted for the Inktomi Bot - "Slurp," it will ensure you are properly indexed. They don't give any guaranteed ranking results, just a guarantee that the site will be visited.

Understanding the concept of pay for inclusion is important for not just web site operators and webmasters, but also to search engine users. There is some fear these programs mean web sites won't get listed unless they pay or that the quality of search results will be degraded. Based on our tests, whether a page ranks well for a specific query still depends on if the proper information is presented to the search engines. A page lacking in content will not show up any higher simply by paying for inclusion in the listings. However, a page with content relevant to a search query will get better noticed.

In contrast, pay for placement auctions, such as used by GOTO does guarantee top search engine listings. If you pay enough, which sometimes means over $5.00 per click, your web site will be the top ranked site for the term(s) of your choice. The obvious drawback here is that while you get to the top, your site will only stay there as long as you can afford it. And that can be a very expensive and fleeting adventure. Acquiring and maintaining a top ranking with a pay for placement search engine such as GOTO involves ongoing expenses because you always pay by the click for any traffic you receive. If you have a popular term, you can go through thousands of dollars a day.

You better make sure that the term you choose has a clear tie to what you sell, and that you sell enough at enough of a margin, so you can pay for it. The nice thing in this system is, it's as close to targeted traffic as you can get. In contrast, the pay for inclusion model has no such guarantee that you'll get any traffic at all. As a side note, Metamend clients who wish to buy keywords on GOTO should visit the partners page, and click on the GOTO logo; you will get a discount on some of the fees.

Another model is followed by Looksmart. Unlike Inktomi, LookSmart is a destination portal. It gets lots of users visiting its search engine. Like Inktomi, it also supplies data to many other search resources. Also Like Inktomi, it is facing pressure from advertisers wanting to see a clearer Return on Investment from it's pay for inclusion model. Looksmart of course has the most expensive model on the market, with prices reaching $299.00 USD for an express listing. Plenty of sites use LookSmart's less expensive (and slower to index) submission programs (also paid) to get listed and still rank well. That's because just paying the entrance fee, doesn't mean you know how to stand out. Getting in the door for a meeting is important, but not having anything to say won't help you stand out. You need to be properly prepared and promoted (optimized) to succeed.

So, if this is a summary of where the search engines sit today, what's to come? One thing to remember is that all of these changes in the models the search engines follow are intended to benefit both web site operators and searchers. Don't forget the search engine industry is only 6 years old or so - It's barely out of kindergarten. I expect that many of the revenue streams used today, will be gone within 12 months.

The one constant in the search engines role has to be relevance. They can only charge money to advertisers and to the companies they supply data to, if their results are justifiable. They have a ROI to justify. Spending any time using a search engine that returns irrelevant results guarantees you likely won't use it in the future. The search engines and directories will continue to evolve, but will have to constantly find ways to improve the relevance of their results in balance with ways to generate revenues. I do not foresee that we will see a pullback on the amount of crawling that the search engines do. In fact, I expect it to get deeper and more refined. I do think that many of the search engines might become more similar to the familiar Yellow Pages - but web sites will have listings in hundreds of categories, as opposed to one or two. This type of service is mostly familiar and efficient, and very time and cost effective to operate, and very useful when people are searching for products and services. But not for all types of searches. The battle will be between allowing listings to be dominated by advertisers' displays, and users wants and needs, and keeping the difference obvious to all. The battle for search engine operators will be to keep up with the different strategies, and keep their sites optimized properly for inclusion under all the various models.

Finally, lest anyone wonder, at Metamend, we do submit to DMOZ, FAST, and all the major and many lesser and virtually unknown search engines on your behalf. We don't like to publish lists, because we constantly add new ones and delete others which stop operating. Where necessary, the submission is set up and done through a manual process, within the proper category. We have been receiving more and more demand for us to perform some of the paid, manual, submissions on behalf of our clientele. To meet this demand, we'll be announcing some new offerings in the next issue of the Mender to deal with the various issues and opportunities. If you have any requests, please feel free to send us your suggestions for new paid services you would like to see added, to: [email protected]

Other articles from this issue:
- Building a Web Site is Like Building a Guitar
- SSH or Telnet?
- The Tortoise or The Hare?

Sign Up Below!

If you would like to receive upcoming issues of The Mender, please enter your email address below and we will automatically add you to our mailing list.

Email address: