Search engines • Search engines are the means by which most people search the Web. • Common examples are Google, Altavista, Direct Hit. Yet they don’t search the Internet • Yet a search engine does not actually search the Web during your search. • A search engine searches itself. • It’s a three-step process. 1) Bots index words • Search engines continually send out hundreds of “robots” or “bots” (or “spiders” or “crawlers” ) • Bots visit web sites, read word by word, and then index those words. 2) A database is created • A huge database of Web sites thus is gathered and indexed by word. • These databases can be huge, with millions of links. 3) The Interface gives you access • Using the keywords you give it, a search engine then searches its own current index. Interfaces are based on rankings • Search engines return results based on a ranking system. • Ranking is the order that files are listed when they are retrieved. The ranking system is secret • These systems are proprietary and often “secret.” In general: • Altavista ranks web pages higher if your search terms are found in the first few words of the page • Google ranks by document “popularity” with other similar searches • Direct Hit ranks by the length of time other users spent at the site Not even half the Web • With all of this software and sophistication, even the best search engines cover only 40-50% of the Web. • And they miss much else on the Internet. Bots hit and miss Bots miss: XML pages, pdf files Dynamically created HTML pages Frames-based pages New pages or recent updated text Some say the Invisible Web is 500 times larger than Web Subject Directories • A subject directory is also a database of web sites and references. • But a subject directory is organized not by keywords but by category or subject. Yahoo! • Yahoo! Is the most popular subject directory. • www. about.com takes the idea a step further with subject guides for selected topics. Subjects are organized by people. • Information is selected, organized and cataloged by a person, not software. • You can usually be more assured that the search results will make sense. You get an index of sites. • Subject directories will not often provide you with ranked web sites. • Instead, you will get a broad index related to your topic, divided further by subheadings. Use for early searching. • Use a subject directory early in your search process to learn about your subject. • You will get fewer links of higher quality. • When you get more specific questions, you should use a search engine.