Crawlers (or bots) are used to assemble information obtainable on the web. By using website navigation menus, and studying inner and exterior hyperlinks, the bots begin to know the context of a web page. Of course, the words, pictures, and other knowledge on pages additionally help search engines like google and yahoo perceive what they’re about.