Web Matrix: CMU Lycos

CMU Lycos (tm): The Catalog of the Internet

The word Lycos is derived from the abbreviation of Lycosidae, the latin name for Wolf Spider. The nickname reflects the automated tool used to find and catalog URL's, commonly called a Web Spider.

Key Links

• Big Catalog: http://query1.lycos.cs.cmu.edu/lycos-form.html
• Small Catalog: http://agent6.lycos.cs.cmu.edu/lycos-form.html

• Big Catalog: http://query1.lycos.cs.cmu.edu/cgi-bin/pursuit
• Small Catalog: http://agent6.lycos.cs.cmu.edu/cgi-bin/pursuit

Organization

• Exclusively Searchable Database:
  • Underlying Search Engine: Pursuit
    Pursuit is a C program that uses a disk-based inverted file retrieval system and a sum of weights to score documents.
  • Features:
    • Multiple Keywords (poor man's Boolean OR)
    • Minimum Keywords per Hit (poor man's Boolean AND)
    • Keyword Negation (precede keyword with a hyphen)
    • Forced Exact Match (append a period to keyword)
    • Encourage Suffixes (append a dollar sign to keyword)
    • Specify Maximum Hits and Minimum Score for Search
  • Limitations:
    • No true Boolean support (but it's planned)
    • No Proximity Searching
    • Alpha-numeric keywords, with a minimum of 3 characters
  • Pursuit scores words by how far into the document they appear; thus hits in the title or first paragraph are scored higher.
  • Search results include match score, document timestamp, length, number of links, document title, headings, and sample extract.
• Forms and non-forms interfaces available, but many search parameters require forms.
• The content is not restricted in any way, except to eliminate overwhelmingly common terms such as HTML, GIF, and Web.

Administration

• Automatic Gathering (Web Spider!), also supports user registration
  Lycos's Web explorer is written in perl, with a C program that uses CERN's libwww library to fetch documents. Lycos will not fetch telnet, mailto, news, file, or WAIS type links (which leaves mostly HTTP, gopher, FTP files).
  
• Search Engine Specifics
  • Lycos provides 5 machines for performing searches, but restricts current requests to lessen the processing load on these computers. (You may be pointed to another machine with a lighter load.)
  • Due to the popularity of the service, it is almost impossible to offer reliable service during the hours of peak usage. It is advisable to perform your requests on weeknights between 9pm and 9am or on weekends to avoid server rejection.
  • Successful requests often complete within about 20 to 30 seconds.
  • For more information, you can examine tables and charts depicting:
    • Accesses by Hour
    • Accesses by Day
    • Accesses by Week
    • Searches or Rejections per Day
    • Thoughput by Machine (Very Informative!)
• Quality of Index
  • As of Feb. 28th, the index catalogued over 3.4 million documents.
  • An average of 10,000 documents are being added each week.
  • Since the database is rebuilt weekly, the number of dead links is negligible.
  • Both the basic search page and the forms interface are very straightforward. The latter also provide pointers to a help page describing how to construct a query.
• Additional Services and Features
  • The server provides both a Help page for new users.
  • When a search request is denied, the server indicates the current load of that machine and provides a link to another server with a lighter load.
  • The server permits both URL registration and deletion by users.

Evaluation

• Strengths
• Weaknesses

Example Usage

• Topic
• Keyword(s), Boolean search controls
• Documents delivered
• Output format