Ten. Gzip Compression: Enable Gzip compression with your server to reduce the measurement of data files transferred amongst the server as well as consumer’s browser. txt file is then parsed and will instruct the robot as to which web pages are not for being crawled. Being a online search engine https://www.youtube.com/watch?v=DhNbKpYq6Ro