Subject Guides

Internet search engines are large indexes of Internet resources compiled by computer programs, rather than people. Often called crawlers, robots, or worms, these programs surf the Internet, identifying web pages, and compiling indexes of words found on them. Much larger than subject directory databases, these indexes often cover millions of web pages and include a wider variety of documents.

There have been many changes that have affected the information industry over the past several years, and these changes are most apparent when choosing a search engine. The first search engines were often projects begun on college campuses, then spun off as small private enterprises. The best known industry change has been the transformation of Google from college project to private company to public stock company, but other changes may have as great an impact on the typical web surfer.

Many search engines, including such venerable names as Lycos, LookSmart and Excite, have been taken over by marketing firms. Where once each of these had its own webcrawler scouring the Internet for new sites, they are now little more than Internet billboards which pass search queries along to Google, Yahoo! or Ask.com, rather than develop and maintain their own indexes of sites. Once each had a distinctive set of features, but now the only apparent difference is the selection of advertisements presented to the searcher. Other well-known search engines have disappeared entirely, including AltaVista, Go, InfoSeak, HotBot, an Northern Light.

The first two sites listed here discuss and review search engines, highlighting features and strong points. The third lists a number of search engines used by New York Times reporters.

Notess, Greg R. Search Engine Showdown. Available: http://www.notess.com/search/. Date of Visit: January 24, 2007.

Reviews, statistics, comparisons of search engines, and tips for searching.

Sullivan, Danny. Search Engine Watch. Available: http://www.searchenginewatch.com. Date of Visit: May 28, 2008.

How search engines work, reviews and status reports on search engines, instructions on searching effectively. Includes listings of search engines, metacrawlers, and other kinds of search sites, but these listings haven't been updated in several years.