Ten. Gzip Compression: Permit Gzip compression with your server to decrease the measurement of documents transferred between the server as well as the user’s browser. txt file is then parsed and may instruct the robotic as to which web pages are not being crawled. As a internet search engine crawler https://www.youtube.com/watch?v=0xsCkQ6bbjk