Individuals use DA to find out which Web-sites are reliable inside their subject and to determine where by to acquire superior hyperlinks. txt file is then parsed and may instruct the robotic as to which web pages will not be to generally be crawled. For a internet search engine crawler https://www.youtube.com/watch?v=OeFqbAORPUg