Thread: Help with Robots.txt

Hello

I used the Google Webmaster tool to create a robots.txt file.

I got this;

User-agent: *
Allow: /

Now, if I wanted the bots to not index/crawl my images folder, what would I add?

Also, is my current robots.txt file alright?

The problem with Google Webmaster Tools is that they create and validate a robots.txt file that Google recognizes, but it does not adhere to the standard. The Allow directive is not part of the robots.txt standard. Other spiders may recognize it, but you will not find the Allow directive to be part of the standard specifications.

If you just want to block the images folder, use the following:

User-agent: *
Disallow: /images/

The Web Robots Pages - Hopes this may help you. best of luck.

Originally Posted by TopDogger

The problem with Google Webmaster Tools is that they create and validate a robots.txt file that Google recognizes, but it does not adhere to the standard. The Allow directive is not part of the robots.txt standard. Other spiders may recognize it, but you will not find the Allow directive to be part of the standard specifications.

If you just want to block the images folder, use the following:

Thanks. I have added that to my file

Originally Posted by cuinee

You use the following syntax:

Code:

User-agent: *
Disallow: /images/

That is exactly what TopDogger wrote..