Posted in Search Engine Optimization

How to Allow or Disallow Bots/Robots to Crawl Your Site

What is the Robots.txt File? A sites Robots.txt file is the most powerful file your site will have. This file controls how search engines crawl your site. Every site should have one, even if it is a basic one. The file sits in the root of the site and is the first file a search engine crawler will read before crawling the site. With this file you can control which site crawls can crawl your site, which pages they can crawl and which pages they can’t. Because of the nature of this file, it is highly important you understand the…