Crawlers (or bots) are used to gather data obtainable on the web. By utilizing web site navigation menus, and studying inside and external hyperlinks, the bots begin to know the context of a web page. Of course, the words, photographs, and other data on pages also help search engines like google and yahoo understand what they’re about. Comply Wit