Defining the Web
The evolution of Web
searching
David Green
The author
David Green is currently the New Media Editor for one of
the ``big five'' professional services firms. He can be
contacted via his Web site at
www.clickmedia.freeserve.co.uk
Keywords
Information retrieval, Electronic publishing, Internet
Abstract
The interrelation between Web publishing and
information retrieval technologies is explored. The
different elements of the Web have implications for
indexing and searching Web pages. There are two main
platforms used for searching the Web ± directories and
search engines ± which later became combined to create
one-stop search sites, resulting in the Web business
model known as portals. Portalisation gave rise to a
second-generation of firms delivering innovative search
technology. Various new approaches to Web indexing and
information retrieval are listed. PC-based search tools
incorporate intelligent agents to allow greater
manipulation of search strategies and results. Current
trends are discussed, in particular the rise of XML, and
their implications for the future. It is concluded that the
Web is emerging from a nascent stage and is evolving
into a more complex, diverse and structured environment.
Electronic access
The current issue and full text archive of this journal is
available at
http://www.emerald-library.com
Online Information Review
Volume 24 . Number 2 . 2000 . pp. 124±137
# MCB University Press . ISSN 1468-4527
124
In that constellation of computers known as
the Internet, transmitted data are split into
small ``packets'' which is an exponentially
more efficient utilisation of bandwidth. This,
together with simpler technologies, has
dramatically reduced the cost of electronic
publishing, resulting in the estimated daily
increase of over 1 million Web pages of
information (Clever Team, 1999). However,
despite its uniform interface and seamless
linked integration, the Web is not a single
coherent element. There are two distinct
elements of the Web: the visible and the
invisible. In order to understand the
implications of this distinction for
information retrieval, it is necessary to first
consider how Web pages are produced.
There are two types of Web page: static and
dynamic. Static Web pages have been
manually created by a Web designer, posted
on to a Web server and are available to
anyone or anything that visits the Web site of
which it is a part. Any changes must be made
manually. Dynamic Web pages are created by
a computer using a script (often CGI, Java or
Perl). This script acts as an intermediary
between the user requesting, or submitting,
information on a static Web page (the front-
end) and a database (the back-end), which
supplies, or processes, the information. The
script slots the results into a blank Web page
template and presents the visitor with a
dynamically generated Web page (Green,
1998a). Figure 1 illustrates this process.
Static Web pages provide the same generic
information to everyone, while dynamically
generated Web pages provide unique
information, customised to the user's specific
requirements. Available to everyone, and for
indexing to all search engines, static Web
pages together constitute the visible Web.
This is what researchers at the NEC Research
Institute refer to as the ``publicly indexable
World Wide Web'' (Lawrence and Giles,
1999).
The invisible Web comprises Web pages
with authorisation requirements, pages
excluded from indexing using the robots
exclusion meta tag and information that
resides within databases that will only ever be
temporarily present on the Web as
Received September 1999
Accepted February 2000
 The evolution of Web searching
David Green
Figure 1 Dynamic Web page generation
dynamically generated Web pages (see
Table I).
The first NEC study (Lawrence and Giles,
1998) estimated that the visible Web
contained at least 320 million Web pages in
December 1997, whilst the second study
(Lawrence and Giles, 1999) estimated the
visible Web had risen to 800 million Web
pages, representing six terabytes of text data,
as of February 1999. Owing to its highly
disparate structure and range of data types,
there has been as yet no scientific research
conducted to determine the size of the
invisible Web.
However, most publishers distribute their
data on the Web by integrating huge
databases, often gigabytes in size, with a
front-end search interface. By virtue of its
commercial professionally published origin,
such information is typically of high value and
more highly structured and indexed than the
visible Web. The user's search enquiry will
generate customised, as opposed to generic,
results. Therefore, for professional
researchers, it can be said that information is
increasingly accessed via the Web, rather than
on it.
Nonetheless, the ``visible'' Web constitutes
a significant contribution to the dissemination
of human knowledge, and as the NEC
studies acknowledged ``much of [this]
material is not available in traditional
databases''. It is no surprise that several
surveys such as Nielsen Netratings or Media
Matrix (www.mediamatrix.com) consistently
show that search engines are amongst the
most popular destination sites on the Web.
Table I Comparison of static and dynamically generated
Web pages
Static Web pages Dynamic Web pages
Manually produced Computer generated
Generic information Customised information
Most are indexable Not indexable
125
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
Web directories and search engines
Web directories explained
What is the difference between a Web
directory and a search engine? A Web
directory is:
. a pre-defined list of Web sites;
. compiled by human editors;
. categorised according to subject/topic;
. selective.
Because humans compile Web directories, a
qualitative decision concerning the content on
each listed Web site has already been made.
Consequently Web directories are popular
with Internet users looking for particular
information because they feel that they have a
head start in identifying ``the best of the Web''
for the topic that they are interested in.
In using a Web directory the user can
navigate through the listings or search across
the entire directory (see Appendix). The
major Web directories also license search
engine indexes to provide secondary results
whenever their human-compiled directory
fails to produce matching results to the user's
query. For example, the world's largest Web
directory, Yahoo!, licenses the Inktomi search
index for just this purpose.
As a result of the manual compilation
process, Web sites that have been indexed by
Web directories will remain listed within that
directory, unless, in the highly unlikely event,
they are manually removed. This permanent
presence is not guaranteed for a listing within
a search engine index, thus making a listing
within a popular Web directory such as
Yahoo! highly desirable.
Broadly speaking, any Web site that
comprises several pages of organised links can
be considered a Web directory. Many
individuals, whether experts in their field or
those passionate about a particular subject,
have compiled such sites. One such voluntary
Web directory which has exploded to global
status, becoming a real rival to world-leader
Yahoo!, is the Open Directory. Other Web
directories of specific relevance to information
professionals include:
. Sheila Webber's excellent Business
Information Sources on the Internet
www.dis.strath.ac.uk/business
. Business Researcher's Interests
www.brint.com/interest.html
Search engines explained
When using a search engine, the user is
searching a database of indexed Web sites. All
 The evolution of Web searching
David Green
search engines have three primary
components:
(1) a ``spider'' that examines Web sites;
(2) an index/database of Web site listings;
(3) interrogation/retrieval software.
Search engine spiders
Search engine databases are primarily built up
by ``spiders''. Despatched on an automatic
and frequent basis by search engines, spiders
are programs that search the Web for new
Web pages, index words and/or links on
those pages, and match the indexed words
with the URL of the page on which they
appear.
Search engine index/database
This is the main element of any search engine
± it is what the user interrogates. Once it
could be said that these indexes were built
along similar guidelines, with the location and
frequency of words the primary determining
factors in results relevance ranking. However,
during 1998, a number of new search engine
providers appeared. These companies built
their indexes according to differing criteria.
The Direct Hit index is based on the
``popularity'' of a Web site, the Google index
is based on the number of links between pages
and sites, whilst the Real Names index is a
pay-for service that enables companies to
register keywords to protect their brands and
company identity. Each of these approaches is
discussed in further detail below.
Interrogation/retrieval software
All search engines have their own customised
software to interrogate their databases.
Essentially, though, they operate according to
similar principles: any Web site which
contains words or terms that match the user's
search query will be presented in the list of
results presented on screen to the user.
Ranking each of these matching Web sites by
relevance is determined by algorithms that
analyse the location and frequency of the
user's search terms against this list of
matching Web sites. The nuances of how
these algorithms work varies between search
engines, which is one reason for the different
results that users usually experience when
running the same search across different
search engines. However, a much more
important reason for these search results
differences is that ``the [content] overlap
between the engines remains relatively low''
(Lawrence and Giles, 1999).
126
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
Portalisation
In a previous research paper (Green, 1998b)
in which I examined the relationship between
portals and information providers, I defined
portals as ``those Web sites that are jockeying
for pole position as starting points for the
Internet user's experience''. Unlike today,
where portals are the focal point for carefully
crafted multi-million dollar e-commerce
strategies, portals initially evolved gradually
from the early experience (1994) of three
types of Internet company. Each of these
companies performed a distinct role in the
information distribution chain that defined
the Internet user's experience. The three
types are discussed below.
Internet service/access providers
AOL and CompuServe (which was later
acquired by AOL) recognised that they were
at the bottom end of the Internet user's
``experience value-chain''. They merely
provided connectivity, which was plentiful.
Such a low-value evaluation would translate
into low margins unless something was done.
Search sites
Yahoo!, Infoseek (now Go), Lycos and Excite
all recognised that usage of their services was
transitory: once the user had found what they
were looking for, they would click on the
result and move off to another Web site. Such
narrowly defined usage was not an attractive
business proposition for either investors or
advertisers.
Browser providers
Millions of users were greeted with the home
pages Netscape (Netcenter) and Microsoft
(MSN) as they switched on their PCs each
morning. Most users did not know how, or
could not be bothered, to change the default
setting in their browsers. This behavioural
inertia was primarily responsible for the huge
traffic figures to these two Web sites. Each
type of Internet company had something to
offer the others. Together they controlled
most aspects of the information distribution
chain. All wanted to increase their
``stickiness'' (the length of time each user
spent at their Web site), before clicking off
onto another site. All wanted to attract as
many ``eyeballs''/users as possible.
To achieve these two goals it was
recognised that the user's primary destination
 The evolution of Web searching
David Green
Web site had to offer as much value to the
user as possible. As users were ceding control
of their initial Internet experiences (they still
do), Microsoft, Netscape, AOL, etc. were
able to control users' primary destination
Web site each time the user logged on. As
search sites were the first desired destination
sites of users, these companies licensed search
engine and Web directory providers to
provide search services. Thus emerged the
concept of the portal.
Portals became a huge success. However, in
the rush for users and their attendant dollars,
many search providers began to neglect their
core service ± their search index or Web
directory. From late 1996 until September
1997 the growth of the main search engine
indexes and Web directories was negligible
(Sullivan, 1999a), despite the continued
relentless growth of the Web. The spurt in
growth of most search engine indexes in the
first half of 1996 was primarily attributable to
the arrival of AltaVista at the end of 1995,
with the largest search index at the time. This
period was also marked by a distinct lack of
search technology innovation. Although meta
search engines such as Mamma, Dogpile and
Metacrawler first rose to prominence during
this period, their search functionality was
essentially based on the ``location and
frequency of keywords'' approach that was
developed by the main search engines.
Meanwhile, the distinction between search
engines and Web directories became
somewhat blurred for the user as the search
engines licensed Web directories and vice
versa, whilst AOL, Netscape, Microsoft, etc.
licensed both. This cross-fertilisation resulted
in portals becoming all encompassing search
sites. It was not until the arrival of a second-
generation of search engine providers in 1998
that new approaches to indexing and
searching the Web became available.
Evolution of search technology
Anyone who has ever seen a diagrammatic
representation of the evolution of life on our
planet, as we currently understand it, would
notice that basic cellular lifeforms were
around for a very long time before the
evolution of more complex biological entities.
However, once this point had been reached,
the rapid diversification of life into ever more
organised and intelligent forms occurred in
127
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
ever decreasing timescales. The same can be
said for Web search technology. By focusing
their efforts on e-commerce and portalisation
the first generation of search sites ± the ``big
five'' ± neglected their core search
functionality. While they reigned supreme for
several years, this neglect, and failure to
appropriately adapt to a changing
environment, created niche opportunities
which were soon exploited by new types of
search providers.
Meta search engines
Meta search engines enable the user to search
across several search engines and Web
directories simultaneously. Some of the most
popular meta search engines include:
Dogpile (www.dogpile.com) searches 14
different engines and directories but does not
eliminate duplicates. It was acquired in
August 1999 by search engine GO2 for
US$40 million in stock and a further US$15
million in cash. At the time of acquisition,
Dogpile only had five employees (The Wall
Street Journal, 5 August 1999).
Mamma (www.mamma.com) searches
seven engines but removes duplicates and re-
orders results according to its own relevance
ranking algorithm.
Others include:
. 2Q ± www.2q.com
. Infind ± www.infind.com
. Isleuth ± www.isleuth.com
. Surfy ± www.surfy.com
. Webtaxi ± www.webtaxi.com
Popularity based analysis
The first generation of search engines created
indexes by spidering Web sites, analysing the
location and frequency of words. Web
directories were compiled manually.
Launched in April 1998, Direct Hit
(www.directhit.com) represented a radical
new departure from these approaches, and
dubbed its methodology ``the third way''. The
system was claimed to be user-controlled as
the ranking of results is based on Web sites
that users have visited. Like many of the
second-generation search technologies, it is
not a separate search engine with its own
index that can be accessed directly. Instead it
provides a second-level analysis of search
results where it is incorporated within existing
search engines, one being HotBot.
Prior to licensing Direct Hit, HotBot
returned a list of results based on the standard
 The evolution of Web searching
David Green
methodology of matching search terms with
content on the Web sites in its index. Now,
Direct Hit will run a second level analysis on
the user's set of results. From its database it
will identify those Web sites which are
popular, according to the number of visits
that each Web site has received, and then re-
rank the search results accordingly, with the
most popular Web sites that match your
search term presented first in the list of
results.
However, the popularity of a Web site can
be largely determined by its search engine
rankings and there are all sorts of ways to
manipulate those if you have a good
understanding of how search engines work.
Direct Hit tries to compensate for this by
boosting obscure sites. For example, a Web
site could provide lots of valuable information
about a particular topic but could
nevertheless feature further down the list of
results of search engines. If a searcher has
been tenacious enough to dig down as far as
result number 100 (presumably an
information professional), and click on it,
then Direct Hit's algorithms will give this site
a big boost up the list of results next time it
appears in other searchers' lists of results. If
other users do not click on this obscure Web
site, then it will drop down the list of results
for subsequent searchers, because it did not
prove popular (Green, 1999a). Since its
launch, the company has successfully licensed
its technology to ten search sites including
AOL, HotBot, Lycos, MSN and LookSmart
and is available within Netscape
Communicator 4.5 and Apple's Sherlock
search utility.
Natural language searching
As already discussed, the first generation of
search engines operated by matching the
keywords submitted by the user to the
contents of the Web pages in their databases.
They did not consider the context of the
search terms, i.e. the syntactical relationships
between the search terms and other
vocabulary within their index. Furthermore,
they search for literal exact matches and
therefore fail to consider semantics or use
thesauri (Green, 1999b). Most search engines
also automatically ignore frequently used
words such as ``or'', ``to'', ``not'', etc. In June
1998 there was a major breakthrough in
addressing these limitations. Two new search
engines were launched within weeks of each
128
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
other. Both offered natural language
searching, but adopted different philosophies
in developing their solutions.
Ask Jeeves (www.askjeeves.com) was
launched 1 June 1998. Billed as ``the first
natural language search agent on the
Internet'' it operates by matching a user's
query against a database of 7 million template
questions. If there is no match then the user is
presented with the nearest alternatives from
the database and asked to select the most
appropriate. It will also conduct a metasearch
across AltaVista, Go (Infoseek), Lycos and
Yahoo! It has now been licensed by AltaVista
for its own search site. However, artificial
intelligence (AI) experts have criticised the
company's natural language claims. It was
named after the resourceful butler in P.G.
Wodehouse's novels.
The Electric Monk (www.electricmonk.com)
was launched a few weeks later. This search
service conducts a syntactical analysis of the
query using natural language algorithms.
These algorithms will also make use of
thesauri to consider alternative related words.
The natural language search is then translated
into a complex Boolean query and submitted
to AltaVista. It was named after a character in
a Douglas Adams novel.
Links-based analysis
The first-generation search engines have
focused on building huge indexes with the
goal of answering every possible kind of
general query. They focus on the content of
each specific page they visit with little
consideration of how these pages interrelate
and connect. As already discussed, the
indexing methodologies they use fail to
consider the complexity of human language:
syntax (sentence structure), synonyms
(different words for the same meaning) and
polysemy (different meanings to the same
word).
Links-based analysis attempts to overcome
these problems by examining the relationships
between pages ± the 1 billion or so hyperlinks
that weave the Web together (Clever Team,
1999). By examining how Web pages link
together, links-based analysis offers
methodologies for identifying authoritative
sources of topic-specific information, eliciting
quality, highly relevant results to users'
queries. Not surprisingly, links-based analysis
has quickly gained prominence amongst
Internet users and is attracting a lot of
 The evolution of Web searching
David Green
attention from both computer information
scientists and corporate Internet investors.
Google (www.google.com), like Yahoo!, was
developed by students at Stanford University.
This technology uses a methodology known
as PageRank (named after Larry Page, one of
its creators) to crawl the Web and analyse
how Web sites link to each other. Results are
ranked on importance, i.e. how many other
Web sites link to them. If you, as a Web site
author, have included hyperlinks to other sites
that you deem important, then you have
exercised some editorial judgement. In the
same way that Web directories, such as
Yahoo!, are compiled by editors on a manual
basis, Google seeks to capitalise on the
editorial judgement of millions of Web site
authors on an automated basis.
As a result, of course, it can analyse far
more Web sites than the humans who build
directories such as Yahoo!. In fact, unlike
search engines that become less useful the
larger their index of Web sites becomes,
Google claims to return even better results
with a bigger index. Google also seeks to
capitalise on the accompanying editorial
commentary by processing the text around
each hyperlink (Green, 1999a).
Links-based analysis does feature in the
relevance ranking algorithms of some search
engine providers such as Excite and HotBot.
However, Google is the only search engine
that is exclusively focused on links-based
searching that is currently publicly available
for Web-wide searching. The company
estimates that its index is between 70 million
and 100 million pages, but, through the links
analysis, enables users to reach an estimated
300 million Web pages. Google's
combination of extensive reach and greater
accuracy of results has quickly catapulted this
relative latecomer to top ten status in search
engine popularity. Data released by Nielsen
Net Ratings in August 1999 showed that
Google gained the largest month-on-month
increase in unique audience figures. Visits to
Google increased by a massive 88 per cent
compared to the average of 2.1 per cent for
the other top ten search engines for that
month. Later that month Google signed its
first licensing deal with AOL subsidiary
Netscape, to be the main search provider on
the Netcenter portal.
Clever (www.almaden.ibm.com/cs/k53/
clever.html) came about when a team of IBM
researchers examining search engine
129
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
effectiveness developed a system that was
referred to internally as HITS (Hyperlink-
Induced Topic Search). The project later
became known as ``Clever''.
Related to the scientific citation index (the
study of how scientific papers refer to one
another), Clever examines the hypertext
context of a keyword search. Like Google,
Clever examines hyperlinks and the
surrounding commentary. Unlike Google,
which crawls the Web, Clever first submits
the query to a search engine such as AltaVista,
and then conducts its links analysis on a set of
pages from the results produced by that
search engine ± typically about 200 pages. By
adding all the pages that link to and from
these 200 pages, Clever creates what is called
a root set ± usually between 1,000 and 5,000
pages. Using linear algebraic analysis, Clever
then begins an iterative process of analysing
this root set of results to divide them into two
categories: authorities and hubs (Clever
Team, 1999). Authorities are Web pages
about a particular topic that have lots of links
to them, i.e. they are authoritative sources of
information. Hubs are Web pages which are a
guide to, or list, authoritative sources, i.e. they
do the most citing.
Hubs are similar to portals in that they act
as a jump point for anyone interested in the
particular topic they cover. Unlike Google,
which retains rankings for individual Web
sites in its index, independently of the user's
search query, Clever will always create a new
root set for each query and prioritise each
page according to the context of the
specific search statement. While not yet
available for Web-wide searching, IBM's
research team is currently refining the Clever
search engine and have been experimenting
with Clever to automatically develop Web
directories.
Focused Crawler is another search engine
technology that is being developed by IBM.
However, it is not yet as developed as Clever.
Unlike other search engines (including
Google and Clever) which perform an
analysis after they have crawled through a
collection of hyperlinks, Focused Crawler, as
its name suggests, seeks to identify highly
relevant collections of data to topic-specific
searching by crawling the Web with a specific
goal, ignoring irrelevant sections of the Web.
In other words, it only crawls Web sites of
relevance to the user's query, rather than
identify a subset of relevant Web sites as a
 The evolution of Web searching
David Green
result of an analysis of a larger set of crawled
sites. Focused Crawler crawls the Web guided
by a relevance and popularity mechanism that
has two parts: a classifier that evaluates the
relevance of a Web page to the user's search
query, and a distiller that identifies ``hypertext
nodes that are great access points to many
relevant pages within a few links''
(Chakrabarti et al., n.d).
Newsgroup searching
The Internet delivers two primary benefits:
content and connectivity. Although distinct,
the two are often closely interrelated. Portals
are a perfect example: they represent the
synergistic exploitation of both content and
connectivity to create e-commerce
opportunities. However, while the Web is the
primary repository of human knowledge on
the Internet, it is not the only one.
Newsgroups, where collections of individuals
share their experiences, knowledge and
opinions on a subject of common interest,
constitute an important area of consideration
for information retrieval. The distinction
between the Web and newsgroups is that the
Web primarily represents a large body of
explicit human knowledge whilst newsgroups
primarily represent a large body of implicit
knowledge. Explicit, codifiable knowledge
can help individuals and organisations, but it
is implicit knowledge ± the realm of
experience, creativity and ideas ± that offers
the greatest potential. In an increasingly
knowledge-based information society, it will
be implicit knowledge that will be needed to
successfully exploit explicit knowledge to
create new opportunities and develop
adaptability.
Considering this, the role of specialised
newsgroup search engines will become more
important as individuals use the Internet to
seek out experts (or indeed anyone who is
qualified) to help with their problems. This
prediction is based not merely on a belief in
human altruism, but also on phenomena such
as the emergent sociology of citations on the
Web (Chakrabarti et al., n.d), the explosive
growth of the volunteer-based Open
Directory (see appendix) and the emphasis on
people/expert connectivity in many corporate
intranets.
There are literally thousands of newsgroups
covering all manner of topics. These are
organised in a tree-like structure with eight
main categories: Comp, Rec, Sci, Soc, Talk,
130
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
News, Alt and Misc. Owing to the huge
number of groups available, specialised search
engines have emerged to identify relevant
groups and postings.
Deja News (www.dejanews.com) is probably
the most widely known newsgroup search
engine. It contains a directory of selected
newsgroups which users can browse through
or search for a particular group, topic or
posting. A powersearch facility enables users
to search by author, date, language, plus
options on how the results are displayed.
Reference.com (www.reference.com) is
similar to Deja News, but also enables
searches in Web forums (Web-based bulletin
boards) and mailing lists (where each posting
is sent to your e-mail address). Users also
have the option to save searches.
Liszt's Newsgroups Directory (http://
liszt.com/news) uses Deja News for searching
on newsgroups, but has its own extensive list
of mailing lists and IRC channels. There is
also a directory of newsgroups with
descriptions for most.
Subject-specific indexes
Company information
There are many sites (usually from company
and business information providers) that any
researcher can visit. The amount and quality
of information provided for free varies.
However, all such sites are Web-enabled
versions of commercial databases, rather than
true search indexes. In a test of the ability of
leading search engines and directories to
deliver relevant results for searches on
company names, conducted by the online
industry publication Search EngineWatch,
HotBot and Google were ranked joint first
search engines while Netscape Search was
ranked first Web directory (Sullivan, 1999b).
However, company research is not the
exclusive focus of these search sites.
Launched in August 1999, 1Jump
(www.1jump.com) is a specialised search
index that focuses exclusively on information
and news about companies. In addition to
providing news, this search engine also
provides details of company executives (titles,
age, background and e-mail addresses),
patents (every patent owned by a company)
and ``peers'' (subsidiaries, parents and related
companies). It also enables the user to visit
other Web pages that are relevant to a
particular company, e.g. an industry
association.
 The evolution of Web searching
David Green
Multimedia and image files
According to industry analyst organisation
Future Image in its report ``Comparative
Evaluation of Web Image Search Engines''
almost 70 per cent of the Web is non-textual.
Considering that humans assimilate and
process information in visual format more
readily than textual format, and the greater
availability of broadband capacity in the near-
future, the role of multimedia search engines
will continue to grow. The three main
specialised search engines in this area are:
(1) Ditto ± www.ditto.com
(2) Scour ± www.scour.net
(3) AltaVista PhotoFinder ±
www.altavista.com
Some other specialised search indexes
include:
. Finding People ± www.whowhere.com
. Law ± www.fastsearch.com/law
. Health ± www.drkoop.com
. Movies ± www.imdb.com/search
. Ask an Expert ± www.vrd.org/locator/
subject.html
. Information Please ± www.infoplease.
com
Search utilities and intelligent agents
As already discussed, meta-search sites such
as Dogpile and Mamma have grown in
popularity as they allow users to search across
different search indexes simultaneously with
duplicates removed and results re-ranked
(depending on the meta-search service used).
Search utilities represent the logical evolution
of this functionality. Unlike meta-search
engines, where the processing power to refine
results still remains on the server the user is
interrogating, search utilities are programs
that are installed on to the user's hard drive.
By shifting processing power away from the
server, and on to the user's own desktop,
search utilities offer a much greater range of
search and results analysis functionality.
Like several of the second-generation
search technologies that have emerged
(Electric Monk, Google) many of these search
utilities incorporate intelligent agents (or
bots). Indeed, many of the powerful features
offered by search utilities, such as language
independent searching, filtering, automatic
refinement of results and document
summaries, active hyperlinking of query
131
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
words and live highlighting of search terms
are possible because of the nature of
intelligent agents. Unlike a standard software
program that will execute specific functions
within clearly defined parameters, agents/
bots:
. are adaptive ± they can interpret
monitored events to make appropriate
decisions;
. are self-organising ± they assimilate both
information and experience;
. can communicate with both the user and
other bots (Green, 1999c).
Agents can search across a wide range of
document types and formats. They can
provide a uniform interface for search queries
across different sources and are true
``infomediaries'' in that they can identify and
search appropriate resources that may or may
not be known to the searcher. The adaptive
element of intelligent agents is central to the
functionality of many search products that
incorporate agents. The following popular
search utilities, which all contain agent
technology, are available as free downloads
and as more comprehensive paid versions:
Mata Hari (www.thewebtools.com) can
learn one set of power search commands and
then automatically translate these for each
search service/database that it queries for the
user.
BullsEye Pro (www.intelliseek.com)
incorporates 11 different intelligent agents,
including technology from Verity to conduct
what it calls ``Web mining''. The different
agents are used to target specific types of
information such as business news in over 450
sources on both the visible and invisible Web.
It will automatically run searches, allows
import/export of searches to other users,
whilst users can chose to receive change alerts
by HTML e-mail, pager or other hand-held
data devices.
Copernic (www.copernic.com) can translate
a search statement for different services and
then simultaneously submit the query to these
search engines, Web directories and
databases. There are also about 20 categories
such as business and finance, science, etc.,
with predetermined Web sources to search in.
Recognising the advantages offered by
search utilities, some search providers have
released a variety of free basic search utility
programs as ``plug ins''. As the name suggests,
once installed, they are incorporated within
 The evolution of Web searching
David Green
the user's Web browser and enable the search
engine provider to offer more features. Search
providers that have released search utilities
include Infoseek (Infoseek Express), AltaVista
(AltaVista Discovery) and more recently
Lycos (See More).
A common function of agents is that they
allow the user to specify a high-level goal
instead of issuing explicit instructions, leaving
the ``how'' and ``when'' decisions to the agent.
This, combined with their ability to search
across data in unstructured formats, to
automatically learn and adapt to user
preferences and to identify patterns, is giving
agent technology an ever increasing role in
Web searching.
XML
HTML is dead. Since XML was completed
by the World Wide Web Consortium (W3C,
the body responsible for developing technical
standards for the Web) in early 1998, it has
attracted an almost evangelical response.
Most Web pages are currently produced in
hypertext mark-up language (HTML). While
HTML's ease of use fuelled its widespread
adoption, it is somewhat limited in that it is
primarily concerned with the design/layout of
a Web page, rather than the information that
actually appears on that page. Considering
that a primary use of the Web is for
information retrieval this design focus is
something of a drawback. HTML is a spin-off
from SGML, a much more robust mark-up
language that was approved by the
International Organisation for Standards
(ISO) in 1986. However, SGML is too
complex for the Web. Seeking to address the
limitations of HTML, the W3C developed
XML as a subset of SGML that would
address the semantic and structural
considerations of information retrieval and
exchange that would work on the Web.
XML is an open technology that offers
tremendous possibilities for electronic
publishing, e-commerce, information retrieval
and data exchange. It consists of rules that
enable anyone to create their own mark-up
language. XML describes information using
pairs of tags that are nested inside one another
to multiple levels (Bosak and Bray, 1999).
These create a tree structure of nested
hierarchies. This convention allows users
direct access to just the particular segment of
the information that they are interested in; for
132
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
example, hyperlinks can go through to the
relevant section of a document rather than the
entire document. It also enables powerful
structured searching akin to database field
searching, but on textual Web pages. In other
words, XML not only enables explicit
description of Web page content, but also
describes the rules for manipulating each data
set contained within the information. This
enables a small program such as a Java script
to process the information on the user's local
hard drive according to their requirements,
rather than the user requesting a new Web
page from the central server. Multiply by
millions of Web users, and this capability will
dramatically decrease the demands on Web
servers and improve network traffic (Green,
1999d). Based on open standards, XML will
allow data exchange between different
computer systems regardless of operating
system or hardware.
As XML is also based on Unicode, a
character encoding system that supports the
intermingling of text in all of the world's
major languages, it will also allow the
exchange of information across national and
cultural boundaries (Bosak and Bray, 1999).
Using various XML style sheets (XSL)
publishers will also be able to automatically
redesign their content for various devices.
There are even style sheets that will read the
text of the Web page aloud, which is of great
benefit to the visually impaired.
However, while XML will deliver great
benefits for searching, publishing and
exchanging information, these benefits will
not be realised without some effort.
First, each industry will need to agree on
standards for the tags used to describe
information that is specific to their discipline.
Mathematicians, genealogists and chemists
have already agreed on standards to facilitate
the realisation of XML's benefits. In other
areas, standards are yet to be agreed upon and
there will be struggles over who controls the
standard (``XML and search'', n.d).
Second, Web publishers will require greater
sophistication than simply knowledge of
HTML, graphics and a few other
applications. They will need new XML tools
and computer programmers and information
scientists who will be able to interpret the
content of the information being published
and provide the appropriate data trees/nested
hierarchies, hyperlink structures, meta data,
style sheets and document definition types
(DDTs).
 The evolution of Web searching
David Green
Finally, search engines will need to learn
the standard tag structure that has been
agreed by each industry/interest group. They
will also need to change their search interfaces
to offer users the choice between text
searching and field/tag searching. Currently,
text-based search engines will return a list of
documents that will contain some
information relating to the user's request.
XML enabled query-searching, like any other
query language, will return the relevant data
that has been extracted from a document,
rather than the entire document. Such query-
based searching can also be used to perform
computational analysis and manipulation of
presentation on retrieved data items (``XML
and search'', n.d).
To facilitate the transition to XML, in
August 1999 the W3C released a hybrid of
HTML 4.0 and XML (XHTML 1.0) for
review. It is highly unlikely that there will ever
be an HTML 5.0. Earlier in April, IBM
launched the Internet's first search engine
that is exclusively focused on XML data,
called xCentral. This search engine is
available from IBM's XML Web site.
The Future
Micro-payments for searches?
Jakob Nielsen predicts ``. . . that in the future
we will have micro-payments for search.
Realistically, to provide quality information
over the long term requires serious effort.
Companies have to be compensated for
providing that'' (Janah, 1999). If users are to
be charged micro-payments then they are
going to start demanding better refinement
technologies for their searches. Much of the
search technology innovation over the last 18
months has come from second-generation
search companies. By focusing on
portalisation and e-commerce the first
generation of search firms have ceded control
of technological innovation to their newer
equivalents.
According to data from
PriceWaterhouseCoopers and research firm
IPO monitor, in the last year search engine
companies have raised more than $274.7
million in private funds and another $282
million in public offerings. Almost all of these
funds are going to this second-generation of
search firms (Investor's Business Daily, 1999).
Searching outsourced?
It is interesting that none of the second-
generation search companies has adopted the
133
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
portal model so beloved of their predecessors,
the ``big five''. Also, if the main portals wish to
introduce micro-payments for searches, these
second generation search companies will
provide the refinement technologies. In the
evolution of Web searching this has created a
symbiotic relationship between the two
generations: to succeed in attracting as many
users as possible and to generate as many e-
commerce sales opportunities as possible, the
first-generation of search firms will continue
to focus their efforts on portalisation and
e-commerce. However, they will need the
new search technology offered by second-
generation firms to provide consumers with
the search requirements they demand ±
search requirements that in turn will fuel
e-commerce consumer buying. The second-
generation search firms need the portals to
attract the consumers who will use their
search services.
Taken to its logical conclusion, it is quite
possible that one or more of the ``big five''
search portal firms will drop out of the
sporadic yet ongoing search index size war.
Instead, they may decide to contract out all
search functionality to second-generation
firms whose core focus is providing better
search technology. By co-opting the Open
Directory, and relegating the results from its
own index as secondary to those from the
Open Directory, Lycos has hinted at the
shape of things that may yet come. However,
until the market matures somewhat, the ``big
five'' first-generation search portals may feel
uncomfortable about completely
relinquishing control of search functionality.
Instead, they may develop their existing
relationships with second-generation firms
into an outsourced/partnership model with
clearly defined service level agreements, etc.
Such a strategic realignment of their business
operations would be in line with current
business process outsourcing (BPO) trends
and would prove popular with their
institutional investors.
Portals go mobile
Using wireless application protocol (WAP),
search sites and publishers alike will be able to
extend their reach beyond the PC to mobile
phones and other hand-held devices such as
PDAs. One such portal already launched is
Zingo, which has been jointly developed by
Lucent Technologies and Netscape. Aimed at
telecommunications providers, Zingo also
enables HTML pages to be converted into
 The evolution of Web searching
David Green
VXML (voice extensible mark-up language)
for audio applications on hand-held devices.
Coupled with the reduced broadband
demand that XML promises to deliver, the
future of information retrieval can be
anywhere you need it.
Search engine standardisation
Launched by Danny Sullivan of
SearchEngineWatch
(www.searchenginewatch.com), the Search
Engines Standards Project aims to foster
standards amongst the major search services.
Participants include representatives from the
largest Web search engines, academics and
industry analysts. Some of the common
standards that the project has helped to
develop include a common syntax for the
command to narrow a search by a specific
Web site, and the ability for all major search
sites to locate an exact URL within their
indexes using the URL: command. Future
proposals include additional commands for
searching and meta tags for controlling search
indexing robots.
This voluntary initiative parallels voluntary
efforts to develop standardised XML tag sets
for specific industries and interest groups. It
would appear that the connectivity provided
by the Internet is also encouraging greater
collaboration in general. These and other
collaborative efforts (such as the Open
Directory) represent admirable attempts to
create a degree of order.
Order vs. chaos
The tension between these two diametrically
opposing forces can be witnessed on the
Internet. The relentless growth in activity is
reducing the Web into a state of digital chaos.
Against this are the commendable efforts of
paid indexers and volunteers attempting to
create an ordered structure. Could the Web
prove to be self-organising just like any other
biological system that evolves to greater
complexity and organisation? In the 9
September 1999 issue of Nature, two
surprising research papers were published
(Albert et al.; Hunerman and Adamic, 1999).
Mathematicians had expected the Internet to
follow the model of random inanimate
networks, but both studies discovered that the
Internet did indeed appear to be ``evolving''
and that its growth resembled organic life.
The Internet is evolving according to the
universal ``power principle'' of physics. This
power principle governs the order found in
134
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
many things ranging from plants to galaxies.
Knowing this will help search engine
providers develop better algorithms that
exploit the predictive behaviour of systems
governed by the power principle. Now
emerging from its nascent stages, the Web
may evolve into a highly organised, vastly
diverse and complicated system.
References
Albert, R., Jeong, H. et al. (1999), ``Diameter of the World-
Wide Web'', Nature, 9 September.
Bosak, J. and Bray, T. (1999), ``XML and the second-
generation Web'', Scientific American, May.
Chakrabarti, S., Van den Berg, M. and Dom, B. (n.d.),
``Focused crawling: a new approach to topic-specific
Web resource discovery'', www.almaden.ibm.com/
almaden/feat/www8
Clever Team (1999), ``Hypersearching the Web'', Scientific
American, June.
Green, D. (1998a), ``Search insider'', Information World
Review, Vol. 14, 1 November.
Green, D. (1998b), ``First through the portal: the business
potential of highly trafficked Web sites'', Business
Information Review, Vol. 15 No. 3.
Green, D. (1999a), ``Search insider'', Information World
Review, Vol. 14, 6 April.
Green, D. (1999b), ``In search of success'', The
Independent, 29 March.
Green, D. (1999c), ``Search insider'', Information World
Review, Vol. 14, 7 May.
Green, D. (1999d), ``Here come the X Files'', Information
World Review, February.
Hunerman, B.A. and Adamic, L.A. (1999), ``Growth
dynamics of the World Wide Web'', Nature, 9
September.
Investor's Business Daily (1999), ``Computers and
technology ± investors betting on big hit in new
Web search engines'', Investor's Business Daily, 2
August.
Janah, M. (1999), ``Web directories profit motive
complicates searches by consumers'', San Jose
Mercury News, 16 August.
Lawrence, S. and Giles, C.L. (1998), ``Accessibility of
information on the Web'', Science, Vol. 280, April.
Lawrence, S. and Giles, C.L. (1999), ``Accessibility of
information on the Web'', Nature, 8 July.
Sullivan, D. (1999a), ``Search engine sizes'', Search Engine
Watch, September, www.searchenginewatch.com/
reports/sizes.html
Sullivan, D. (1999b), ``Company names test'', Search
Engine Watch, August, www.searchenginewatch.
com/reports/names.html
``The top ten referring search engines'' (1999), September,
www.statmarket.com
[The] Wall Street Journal (1999), ``Web-search firm
acquired in $55 million transaction'', The Wall Street
Journal, 5 August.
``XML and search'', Search Tools, www.searchtools.com/
related/xml.html
 The evolution of Web searching
David Green
Appendix. Company profiles
AltaVista
Launched by Digital Equipment Corporation
in December 1995 as the largest search
engine on the Web. The launch of such a
large index forced all the other major search
engines to increase the size of their own
indexes during 1996. Has consistently
remained as one of the largest search engines.
This, combined with its range of powerful
search commands has ensured its popularity,
especially with researchers. In 1998 Digital
Equipment Corp. was acquired by Compaq
who spun off AltaVista as a separate company
in January 1999. Later that year in June,
Internet investment company CMGI (which
also has a shareholding in the Lycos Network,
who in turn are owners of HotBot), acquired
a majority 83 per cent stockholding in
AltaVista. CMGI has announced plans to add
new services to AltaVista including an
updated index that refreshes at least every 28
days. A multimedia search engine has already
been added. CMGI plans to publicly list
AltaVista at some point in the future.
www.altavista.com
Ask Jeeves
Launched in June 1998 as ``the first natural
language search agent on the Internet''.
Operates by matching a user's query against a
database of 7 million template questions,
presenting variant questions if there is no
match. It will also conduct a metasearch
across AltaVista, Go (Infoseek), Lycos and
Yahoo! Has been licensed by AltaVista for its
own search site.
www.askjeeves.com
Direct Hit
Launched in April 1998. Offers co-branded
search solutions to other search engine
providers. It operates by providing a second
level ranking of the user's search results on
the basis of ``popularity''. The company
currently licenses its technology to ten search
sites including AOL, HotBot, Lycos, MSN
and LookSmart. In August 1999 the company
announced that it had raised almost $27
million from venture capital firms and private
investors.
www.directhit.com
Excite
Launched in late 1995. This search engine
was immediately popular with users due to its
135
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
large index and integration of non-Web
material such as company information. The
company acquired two of its competitors,
Magellan and WebCrawler during 1996. In
January 1999 Excite was purchased by high-
speed cable Internet access provider @Home
and the company became known as
Excite@Home. In June 1999 the company
lost its licence to provide the search results at
the AOL NetFind portal to competitor search
engine provider Inktomi. Later in September
the company announced the launch of a huge
250 million page index and powerful new
search functionality.
www.excite.com
FAST
Launched in May 1999 with the largest ever
search engine index at the time ± over 200
million pages. This Norwegian company
ambitiously aims to index all of the Web ±
hence its URL. Unlike other search engine
companies, who use mainframe computers to
power their services, FAST has linked
together a few hundred Dell PCs (Dell has a 5
per cent stake in the company) and uses
parallel processing to deliver its service. The
company plans to have increased its index size
to 300 million pages by the end of 1999.
www.alltheweb.com
Go (Infoseek)
The Infoseek search engine was launched in
1995. The Disney Corporation acquired a
large stake in Infoseek in June 1998, and in
January 1999, Infoseek, was re-launched and
re-branded as a portal site known as Go. Like
many other search portals, Go offers users the
option of searching the index or browsing
through a human-compiled Web directory.
www.go.com
Google
Launched in 1998. Developed by students at
Stanford University (as was Yahoo!), Google
focuses on the link structure of the Web to
determine relevant results for the user. Its
proprietary technology, PageRank (named
after co-founder Larry Page), crawls the Web
analysing both the links between Web sites,
and the accompanying text around each
hyperlink. The company estimates that its
index is between 70-100 million pages, but
through the links analysis enables users to
reach an estimated 300 million Web pages ±
which is currently a much greater reach than
 The evolution of Web searching
David Green
any other search engine provider. Like most
other second-generation search providers, the
company is focusing on co-branding its
technology rather than building its own
search portal. In August the company signed
a deal with AOL subsidiary Netscape to be
the main search provider on the Netcenter
portal.
www.google.com
HotBot
Launched in May 1996 by Wired. Acquired
by Lycos in October 1998, but continues to
be run as a separate service from the Lycos
search engine. Accesses the Inktomi search
engine index, rather than compiling its own
index. However, primary results are derived
from Direct Hit, the popularity-based search
provider (see above). Directory listings are
derived from the Open Directory (see below).
www.hotbot.com
Inktomi
Founded in February 1996, Inktomi is
probably the most famous search engine
index. It powers the search results for several
famous portals and search sites including
HotBot (where it debuted), Yahoo!, AOL,
MSN Search and SNAP. However, not all of
these companies access Inktomi's full 110
million page index and there are variations in
results between the different search sites due
to the different filtering and relevance ranking
algorithms Inktomi provides to each partner
company. It is not possible to interrogate the
Inktomi index directly.
www.inktomi.com
LookSmart
Launched in October 1996. Like Yahoo!,
LookSmart is a human-compiled directory. In
addition to providing its own search site, the
company also licenses its directory to other
companies including AltaVista (who in turn
provide search results to LookSmart
whenever there is no match to a user's query
within the directory) and in August 1999 with
Excite (replacing Excite's own directory).
During that same month, the company raised
US$92.4 million on its public listing of 7.7
million shares at US$12 each.
www.looksmart.com
Lycos
Launched as a search engine in May 1994.
The company rapidly diversified into other
136
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
areas (AngelFire, Tripod, WiseWire, etc.) and
e-commerce has become its primary focus.
Although it acquired rival search engine
HotBot in October 1998, it switched to a Web
directory format in April 1999. Primary results
are now derived from the Open Directory (see
below), with secondary results coming from its
own index. It has also added almost 8,000
databases of information specific to different
industries. HotBot continues to be operated as
a separate venture.
www.lycos.com
Northern Light
Launched in August 1997. Has continually
been one of the largest indexes, gradually
increasing in size until it became the biggest
search engine (indexing 16 per cent of the
Web). This leading position has since been
superseded by the launch of FAST in May
1999. The company also offers a special
collection of non-Web material such as
newspaper and magazine articles. Whilst it is
free to search within the special collection,
users must pay a charge (up to $US4) to view
any articles from this collection. Search
results are clustered in folders by topic. Like
AltaVista, this search engine is popular with
researchers due to its scope and functionality.
www.northernlight.com
Open Directory
Launched in June 1998. This directory uses
volunteer editors to catalogue the Web. This
initiative quickly gained prominence and was
acquired by Netscape later in November of
that year. Netscape pledged to allow anyone
to use the directory. In April Lycos re-
launched itself as a directory service, deriving
its primary results from the Open Directory.
http://dmoz.org
RealNames
Launched in 1998. Formerly known as
Centraal Corp., RealNames charges
companies an annual US$100 to register
individual keywords, such as company name,
or a brand name. Obviously many companies
want to, and do, register many keywords to
protect their brands etc. This has proved a
very successful economic model for the
company and in August 1999 it successfully
raised over US$70 million from venture
capitalists in a third round of financing.
Although the index is directly available as a
download from the company's Web site, and
 The evolution of Web searching
David Green
is incorporated within Microsoft's Internet
Explorer 5 browser, its most notable success
has been access from search engines that
license its index, such as AltaVista and Go
(Infoseek).
www.realnames.com
Yahoo!
Launched in late 1994, Yahoo! has become
the most popular search site on the Web,
137
Online Information Review
Volume 24 . Number 2 . 2000 . 124±137
accounting for a staggering 43.5 per cent of all
search engine referrals in August 1999 (``Top
ten'', 1999). It is the Web's largest human-
compiled directory, listing over 1 million sites.
These directory listings are also supplemented
by search results derived from Inktomi's 110
million page search index. Launched a new
photo search service during the summer.
www.yahoo.com