Search Engines. Which Should You Be Using?

Our Experience So Far!
Our Experience as a Contributor
"Categories" v. Indexed Searching, Sites v. Pages
Lawrence Chard's Contribution to Index and Date Retrieval Theory
Sample Search Results
Our Ratings Out of 10

Our Experience So Far!

Since we first wrote this page, Google has appeared, and is now our first choice and favourite:-
google.com

We recommend Alta Vista as the best internet search engine, in our experience. If you haven't used it, we suggest you try it at:
altavista.com
Also good is Northern Light, and rate it at least as good as Alta Vista. It just isn't very well known in the UK. Find it at
http://www.northernlight.com

Here is a list of others which you may care to try:-
http://www.alltheweb.com
http://www.everywebsiteintheworld.com
http://search.excite.com
http://galaxy.tradewave.com
http://www.hotbot.com
http://www.infoseek.com
http://www.lycos.co.uk
http://www.mckinley.com
http://ukplus.co.uk
http://webcrawler.com
http://www.yahoo.co.uk
http://www.yahoo.com

There are others often provided by one of the above, but "badged" by your dial up ISP.
Your may have your own preferences, but our opinion has been formed over about a year using the web, and 6 months bringing our own website into being. 24carat.co.uk went "live" on December 1st 1998, but we spent over 6 months before that developing it on a "homepage". Our experience trying to get our site or its various pages listed has been quite educational.

Like most new users, at first I did not know one search engine from another. They nearly all work in slightly different ways, using different logic and syntax. Over a period of time, I noticed that I had started to use Alta Vista as my first choice. At that stage, I could not have explained why apart from being able to use a + or - sign to include or exclude a particular word. Having now been submitting our site to most of the above search engines for some time, my preference has sharpened.
There are two sides to use a search engine. One is searching for information it contains, the other is getting it to accept information such as new pages or sites. Most people's experience is from a user or consumer viewpoint, and if this isn't satisfactory, then the search engine will eventually be discarded in favour of a better one. However, if the search engine does not accept submissions of new data, then its index will eventually become stale or obsolete, and it will be worse than useless, because the consumer may not realise that its information is stale, and that he is being denied access to fresher information. I will attempt to share our experiences with you. First as a contributor, trying to get our site or pages listed. Please note that search engines change with time, and our experience and opinions have changed since we first compiled this page.

Our Experience as a Contributor

alta vista
When we add a new page to our site, we usually add it to AV immediately, and it appears in their index within a few days. Their "Add URL" page claims "Please submit only one URL. Our crawler, Scooter, will eventually explore your entire site by following links.", but this does not seem to work, at least in our experience. After 6 months with our old site, only about 5 pages were listed, all of which we added separately. Searching for "24carat.co.uk" finds over 200 pages, most of the pages we have submitted so far.

everywebsiteintheworld
Recent, but instant. Add a page, and see it immediately afterwards. If they can keep this up as they grow, this could be a world beater. Main drawback is that each page needs to be submitted using their on-line form, which is time consuming. What a snappy and concise domain name!

excite
"Add Your Site to Excite's Database Add your site to Excite's continually refreshed database of URLs and our spider will crawl and index every page contained therein. A powerful program, our spider can index an entire site, not just a page or two. Submit your URL and every two weeks, our spider will call your server, access your Web site, and crawl through every page, gathering information to be indexed and noting any content changes you've made."
This feature doesn't work! We submitted our old site about 6 times in 6 months, and still weren't listed. We submitted 35 different pages, on about 8 different occasions, totalling about 41 pages submitted, between November 1998 and January 1999, and had got 1 page listed by January 1st. It's still the only one listed. Searching for "24carat.co.uk" fails to find it at all!

galaxy
Submitted about 20 times. Searching for "24carat.co.uk" finds 1170 matches, the first of which is "American MGB Association Home Page"!, the next 19 seem just as irrelevant. The 20th does contain"UK" in its title! None of the 1170 are ours, as far as I can tell. They now guarantee to review your site if you pay $25, although they are careful to avoid saying that they will list it.

hotbot
Submitted our old site about 6 times over 6 months, and have now got 1 page listed. Submitted about 35 different pages over 6 occasions since November, and still not listed, despite their very encouraging message after adding an URL, "Got it! Your URL will be added to HotBot's index within the next two weeks.We have e-mail them to ask what we are doing wrong, but it still makes no difference.

infoseek
This was listing 1 page of our old site, after 6 submissions in 6 months. We submitted about 35 different pages of our new site since November, and got nowhere. Their "Add URL" feature often gave us a "denied access" message, adding darkly that there are some large domains from which they do not accept automated submissions, and advising us to e-mail our URLS. We tried this several times, before giving up and e-mailing them also. We got a very prompt and detailed reply, which was helpful. We now have 16 pages listed, and expect to have more soon.

lycos uk
Submitted our old site about once a month for 6 months without any success. Submitted about 35 pages of our new site, over about 6 occasions since November, without any results. Its "Add URL" page encouragingly says " Add a site to Lycos Now it's easier than ever to add or check your site on Lycos. Follow our easy guide and keep us up to date. New to the process? Get help below: FIND YOUR SITE ON LYCOS: You may now use our URL Finder to confirm that your URL is registered with us: Top of Form 1 Enter your URL: Bottom of Form 1 ADD YOUR SITE NOW: Lycos now offers instant spidering of your site. Enter your URL below: Top of Form 2 Enter your URL: Enter your email address (required) Bottom of Form 2 TIPS ON ADDING A SITE Don't bother registering every page in your site individually, unless you're really anxious. (Our spiders will automatically register all additional screens shortly after your home page.) Once you register with Lycos, your site will be spidered immediately. However, it will not be immediately available on Lycos. We receive thousands of URL's every day making us the most comprehensive service available. Therefore, it generally takes about 2-4 weeks to be listed with Lycos.
Update.
We finally got 2 of our sites listed, however they only show up when searching "UK only", they disappear if trying to perform a "world-wide" search. As over 50% of our enquiries are from outside the UK, this is frustrating!

magellan
Owned by Excite, with similar results. Searching for "24carat.co.uk" produces 0 results, but searching for "Chard" finds us in #12 position. Can only find 1 page.

northern light
About two days after publishing the original of this document, I read another internet source document by an American legal researcher relating her experiences. She listed a small number of search engines, amongst which was Northern Light. I had heard of it, and had submitted our site to it once via one the "add your site to 30 search engine" pages. I decided to check it out, see if we were listed, and re-submit one or more pages. Imagine my delight and surprise, when searching for "24carat.co.uk" produced over 100 of our pages, plus a few others relating to us. This was more pages than were being listed by Alta Vista. Apparently, Northern Light's spider must done its job, crawled around our site, listed all the pages, and re-visited periodically, to add more pages.
For this alone, I revised my opinions, and wished that I hadn't given Alta Vista 9, perhaps I will need to revise the scoring in my table, I don't like to give 10/10 ever. I would rate Northern Light as being at least as good as Alta Vista. There is also a bonus, Northern Light creates what it calls "custom folders", and automatically attempts to classify the documents it finds into suitable categories, such as information, for sale, "24carat.co.uk" domain, goldsmithing, gemstones, others. This can't be expected to work perfectly, but it really is an interesting attempt to organise the data into intelligent categories.
Update.
It just took us about 4 months to get another site listed, and we are still trying on the third one, so we revise our opinions of Northern Light downwards.

ukplus
When I first saw UK Plus on the search page provided by our ISP, Globalnet, I thought it was a great idea. After all, one of the main problems I find is that most of the sites listed are American. We submitted at two or three times, with no success. I tried to contact them via their response links on their site. At least two of them didn't work. Eventually an e-mail got through to them, apologising for losing our previous submissions, saying that our site and content were good, and worked well, and listing us under two categories.

webcrawler
This is powered and owned by Excite. Our experience is similar to Excite, except that 0 pages were listed by 6th January, so I e-mailed them to ask why. I got a quick response to say that they have now added our site directly to their list of sites to be "spidered", and it should show up in the next 2 to 3 weeks.

yahoo uk
See yahoo.

yahoo
Probably the most popularly used search engine.
Submitted our old site about 6 times over 6 months with no success. Submitted our new site about 4 times since November, until just before new year, I found a name there, and e-mailed to ask were we doing something wrong. To my surprise I got a quick response saying that our site had been added, and would be listed within 2 or 3 days. It is now listed under 2 categories, including under yahoo UK. Our third site is still not listed, so we keep trying.

"Categories" v. Indexed Searching, Sites v. Pages

When submitting to Yahoo, I found that I wanted to add our site to a large number of categories, because we did not appear to fit neatly into any single 1 category, however some of the categories under which we could be listed would have a lower relevance rating compared with other categories. For example, our main business is split into 3 parts, Coins, Diamond rings, and Gold Jewellery. To complicate it further, we are definitely not an ordinary jeweller, we have a showroom which we do not refer to as a shop, we are not in a prime high street location, we don't sell watches. We don't seek repair business. Because we started wholesaling jewellery before we started retailing it, we don't accept credit cards. We sell to anybody - consumer, retailer, wholesaler, manufacturer. Our direct to consumer prices are too low, often about half the prices we see elsewhere. Yet we don't wish to call ourselves a "discount" jeweller, because it sounds cheap and nasty, and all the "discount jewellers" I have seen have been cheap and nasty. In other words, we are the sort of business which doesn't easily fit into any single category.

What about "Sites v. Pages"?
The search engines which use categories expect to list "sites" rather than "pages", and expect all sites top fit neatly into a category. With our site, we have a humour (in my opinion) page, which concentrates on humour related to coins or jewellery; does that mean our site should be listed under humour? I think not, but a search engine using a category paradigm would have to omit us. A page based index such as Alta Vista can list our humour page without forcing it into a category. Anybody searching for jokes or humour connected to coins, gold, jewellery, Blackpool or Chard, should be able to find this page.
Similarly if anybody wants to find a business which deals primarily with a mixture of coins and jewellery, then the page based approach should find a number of our pages which combine both.

Category searching may have worked well for manual systems, and page indexing would have been impractical manually. With the advent of fast powerful computers, it is possible to build a database index system which can permit very flexible search criteria. This seems to be the great strength of the Alta Vista system. All other systems seem to trying to use a manual category type system stored performed on a computer. In my view, the flexible, page based approach should work best for searchers using their intelligence. It is quite simply a vastly superior method.

Lawrence Chard's Contribution to Index and Date Retrieval Theory

One thing I have not yet seen, and which I believe could provide a major theoretical improvement in indexing techniques, would be an exclude system for contributors.
For those who have never uploaded an html page, or written an internet source page, I will explain. html provides for a "keyword" section near the start of any document. This is intended as a guide for indexing, to be used by search engines. Unfortunately it is open to some abuse. Vendors of widgets could insert keywords like sex, Pamela Anderson nude, etc. to generate more "hits" on their page. Obviously the search engine providers realise this, and have implemented systems to check that keywords reasonably match the page and site content. It is also why some search engines totally ignore keywords, and index on page content alone.
What is missing is a "negative keywords", so that the author of a page could assist the search engine in placing that page more accurately. As an example, if I search for my surname "Chard", hopefully I will find many of our own pages, as I should, I will also find other pages referring to other people called Chard. So far, so good, but I also find (1) hundreds of pages about "Swiss Chard" and other beet / spinach type vegetables, soup recipes, (2) Pages about the famous town of "Chard" in Somerset, and (3) hundreds of pages containing the names Richard as in person, and Chardonnay, as in grape variety or wine. If when writing our pages, we could specify the "negative" keywords Richard, Chardonnay, vegetable, and Somerset, then search results would be improved.
At present, Alta Vista, for example allows the searcher to exclude words by placing a "-" sign in front of them, but this process could be assisted by the document author if he were allowed to use negative keywords.
This would be similar to the logic used within a thesaurus, where to refine a search, one uses one class of meaning of the required word.

Sample Search Results

Recently, an enquirer kindly told us he found us by searching on Yahoo by entering "UK GOLD COINS". I tried it, and we came up in second place behind the Royal Mint, quite successful, not only for us, but for our customer. What the customer actually wanted to locate sources for was "gold sovereigns", and presumably he preferred a UK supplier. As this is a small but distinct sector of our trade, we tried searching for "gold sovereigns". Here is a summary of our findings:-

alta vista
419 matches. First was from an Australian dealer. One of our pages (.../goldsovereigns.html) was second. Most of the first 10 matches looked quite relevant.

everywebsiteintheworld
9 matches, all of which were ours!, 2 of these pages were not directly about sovereigns. All were relevant. Although we could feel pleased about this result, most searchers would presumably prefer to find more choice of author!

excite
555,258 matches. The first was titled (slightly ungrammatically) "Great Britain sovereigns", from an American dealer, obviously relevant. The first 10 pages appeared quite relevant, although half of them were about other gold coins.

galaxy
4 matches. None of which appeared to be relevant. The first result was "The Best of Edward Gibbons"!

hotbot
1715 matches. The first 10 looked suitably relevant, but sadly did not include any of our pages (see above).

infoseek
6,237,997 matches. Of the first 10, only one, the second was directly about gold sovereigns. The number 1 , 4 and 8 slots were taken by a book review or advert about "The Light Princess", the third result appeared to be written in "assembler", the 5th was a hotel. Our ".../goldsovereigns.html" page, titled "Chard - Gold Sovereigns as Bullion & Collectors Coins" ranked in 53rd position.

lycos uk
Over 100 matches (number not specified). First 10 appeared relevant. None of our pages, including our "Gold Sovereigns" page, found.

magellan
26,250 matches. The first 4 looked relevant, including our "...index.html" page in 4th place. Fifth was about "the Gold Coast", the rest were mainly about monarchy.

northern light
Listed 4 of our gold sovereign pages in its first 25, of which 90% appeared relevant, total over 21,000 matches. ukplus
2 matches. Ours in first place, albeit titled "Chard Jeweller", and the Royal Mint in second place. A compact number of matches, but 100% for relevance. Most searchers would presumably prefer slightly more choice.

webcrawler
33,367 matches. Ours listed third out of the first 25, which appeared to be about average for relevance.

yahoo.co.uk
5 matches "Yahoo! Site Matches (1 - 5 of 5)", all of which were for Arts: Humanities: History: Browse By Region: Countries: Australia: Complete Listing (+ 4 other categories) Sovereign Hill - open air museum bringing Victoria's Gold Rush era to life.
So really only one match which was not primarily about gold sovereigns!

yahoo
As for Yahoo UK.

Our Ratings Out of 10

Taking into account relevance, particularly of the first page of results, which is what most searchers view.

Search Engine	Score	Comments
alta vista	9	excellent
everywebsite...	6	limited
excite	5	average
galaxy	0	no comment!
hotbot	4	average
infoseek	5	low relevance
lycos	4	average
magellan	4	average
northern light	8	excellent
uk plus	6	concise
webcrawler	5	average
yahoo uk	4	fair
yahoo	4	fair

Obviously, results for a different set of search criteria could be considerably different, although my own observations and experience have been broadly in line with the results in the above summary.

We would be delighted to hear from Search Engines, Users, and Authors.

All comments welcome!