Search Engine Observations: Return of The Metacrawler

Search engines come in all flavors, sizes and depending upon whom you speak with, different quality levels. With so many search engines out there these days, how do you wade through them? You could simply use the most popular ones such as Google( -Yahoo! ( -or Microsoft’s new search engine ( but because these larger search engines directly compete with each other for market share, you may be getting short changed when it comes to looking for very specific results. These big engines all pride themselves on accurate searches, so you get very similar results from all three. Since they are all very similar, the question that comes to mind is “what happened to the rest of the resources which didn’t make it into the top one hundred?” Good question! This can happen very often, especially if the information you are attempting to retrieve is not commercial in nature, but rather, something much more obtuse, something not popular at all and not sold by Amazon. So what can you do?

In this case, your best bet may be to seek out a Metacrawler search engine. A Metacrawler gathers information from a much wider spectrum of the Internet, by utilizing several (some times dozens) of different search engines before producing its results. As a much broader range is covered, a much larger sampling and cross section of material is accessed before the results are presented.

Not every web site on the Internet can afford professional search engine optimization or can afford to hire a top notch SEO firm to get top ranking. Most academic, scientific, medical and even Governmental databases may not bother with optimization in any serious form, so their respective web sites and pages are unlikely to show up highly in the ‘popular’ engines. A Metacrawler which has access to these secondary or smaller search engine databases, can sometimes offer a better chance of finding those less popular web pages. Look at it this way: Google brags that it archives over eight billions pages(1) within its index. Probably true, but how many pages of results do you have to dig through to find an obscure reference to a non popular subject? Perhaps hundreds. The advantage of a Metacrawler’s approach is access to the rare and obscure, as well as the popular.

Here is a hypothetical example: Let’s say I enjoy gardening and have always been intrigued by the fact that roses come in all colors except black. Black roses are rare. I would like more information on how to grow this variety.

I search for the term “Black Rose” in Google. Within the first ten listings, I was presented with a Gothic Clothing Store, a nonprofit Foundation, a Book Store, a Boston Restaurant, a Florist, An anarchist web site, an acoustic society, and a San Francisco bay area radical health collective.

Notice anything about gardening? Neither did I. My search resulted in zero accuracy (for my specific topic of interest) and more importantly, it was a waste of my time.

Now let’s try the same thing on a Metacrawler. I went to Dogpile, and typed in the same search term. Along with very similar (popular) results such as those above, I am also presented with some helpful links on the right hand side of the page, which assist me in narrowing my search parameters. Since the search phrase ‘black rose’ is (purposely) somewhat vague, the Metacrawler attempts to help me refine the term. It asks me “Are you looking for: Black Magic Roses? Black Silk Roses? To buy Black Roses or to find out about blooming Black Roses?” Since my intention was finding a gardening topic, I choose the “Blooming Black Rose” option. Click. Eureka! I am presented with the “Rose-Chat Gardening Club”, as my number 3 result. I am now on my way to discovering the information I was seeking with very little time spent digging, if you’ll excuse the pun.

One interesting aspect of this test example is the Metacrawler found the result I was seeking by scanning both Google and Yahoo! databases. That’s right, the result I was seeking was indeed listed within both of the big engines, but neither one presented it to me as easily and quickly as did the Metacrawler.

Run a couple of your own tests and you may find that the Metacrawler is a long way from obsolete. In my humble opinion, they are on the rise again. Here are several Metacrawlers for you to try out. Have fun and never count out the little guy.


IXquick: ( scans a dozen engine databases)

EntireWeb: (good European Metacrawler)