Guruji'S Web

Robots.txt is pretty easy to use, to create you just need to use notepad and follow the right syntax. Luckily this is also very similar and is comprised of only a few elements.

The file is made up of several records and each record features a user-agent section and a disallow section. You may choose to include comments with the use of a “#” at the start of a line.

The “user-agent” section defines which robots should follow the command. To list multiple robots simply use multiple user-agent lines. You can also use the wildcard character “*” to force ALL robots to obey the following command. An example of this line is below.
User-agent: *

The “Disallow” section is used for specifying the directory or file that should not be accessed. It is fairly simply sytax and you just need to include the directory (excluding you base URL) or file similar to the following example.

Again you can have multiple lines to disallow a selection of files or directories.
Disallow: /folder1/

The “allow” command can also come in useful, it lets you specify a specific file to allow in a directory you may have disallowed. It still needs to be paired with a User-Agent command but can be added in with disallow commands. Here is an example of a full record using all of the above.
User-Agent: *
Disallow: /folder1/
Disallow: /folder2/
Disallow: /folder3/
Allow: /folder1/important.html
Allow All

Happy with all the robots getting to your site? Just use this simple code to open the gates and ensure you’re not blocking anything
User-Agent: *
Disallow:
Block All

Site down for a while for construction? This will block everything form spidering. Don’t forget to remove it when you’re done!
User-Agent: *
Disallow: /

Linking to your Sitemap

A handy little trick that some robots allow is to link to your sitemap inside your robots file. Simply use the following example and modify the URL to suite your own domain.
Sitemap:http://www.yourdomain.com/sitemap.xml