txt file is then parsed and can instruct the robotic regarding which pages will not be to be crawled. To be a search engine crawler could retain a cached duplicate of the file, it could occasionally crawl pages a webmaster does not wish to crawl. Webpages generally prevented from becoming crawled in