Robots Exclusion

5th Feb 2004

Web Server Administrator's Guide to the Robots Exclusion Protocol. This page gives you a run-through of what to include in your robots.txt file. Here's a brief summary:

To exclude all robots from the entire serverUser-agent: *Disallow: /To allow all robots complete accessUser-agent: *Disallow:Or create an empty "/robots.txt" file.To exclude all robots from part of the serverUser-agent: *Disallow: /cgi-bin/Disallow: /tmp/Disallow: /private/To exclude a single robotUser-agent: BadBotDisallow: /To allow a single robotUser-agent: WebCrawlerDisallow:User-agent: *Disallow: /