Robots.txt is usually a file that exists on the basis directory of each website and may be used to instruct search engines like yahoo on which directories/data files of the website they could crawl and include inside their index. Whenever you use a search engine, you’re in search of content https://news.rafeeg.ae