To discover content on the Internet, search engines rely heavily on automated software. There are several different names of these automated software agents: spiders, crawlers, and bots.

A search engine bot finds websites and scans them by following links to web pages. Googlebot has the highest exposure, accounting for approximately 65 to 70 percent of the market share, but other search engines do have crawlers.