Posted in Search Engine Optimization

How to Make Effective Use of robots.txt for Better SEO

Restrict crawling where it's not needed with robots.txt A "robots.txt" file tells search engines whether they can access and therefore crawl parts of your site (1). This file, which must be named "robots.txt", is placed in the root directory of your site (2). User-agent: * Disallow: /images/ Disallow: /search (1) All compliant search engine bots (denoted by the wildcard * symbol) shouldn't access and crawl the content under /images/ or any URL whose path begins with / search. (2) The address of our robots.txt file. You may not want certain pages of your site crawled because they might not be useful to users if found in…