Results 1 to 5 of 5

Thread: Help with Robots.txt

  1. #1

    Question Help with Robots.txt

    Hello

    I used the Google Webmaster tool to create a robots.txt file.

    I got this;

    User-agent: *
    Allow: /
    Now, if I wanted the bots to not index/crawl my images folder, what would I add?

    Also, is my current robots.txt file alright?

  2. #2
    The problem with Google Webmaster Tools is that they create and validate a robots.txt file that Google recognizes, but it does not adhere to the standard. The Allow directive is not part of the robots.txt standard. Other spiders may recognize it, but you will not find the Allow directive to be part of the standard specifications.

    If you just want to block the images folder, use the following:

    User-agent: *
    Disallow: /images/
    "Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote." -- Benjamin Franklin


  3. #3
    The Web Robots Pages - Hopes this may help you. best of luck.

  4. #4
    Quote Originally Posted by TopDogger View Post
    The problem with Google Webmaster Tools is that they create and validate a robots.txt file that Google recognizes, but it does not adhere to the standard. The Allow directive is not part of the robots.txt standard. Other spiders may recognize it, but you will not find the Allow directive to be part of the standard specifications.

    If you just want to block the images folder, use the following:

    Thanks. I have added that to my file

  5. #5

    Smile

    Quote Originally Posted by cuinee View Post
    You use the following syntax:

    Code:
    User-agent: *
    Disallow: /images/
    That is exactly what TopDogger wrote..


Similar Threads

  1. The EPFL mini-robots
    By kiki in forum General Chat
    Replies: 1
    Last Post: 2 June, 2010, 02:24 AM
  2. robots.txt help
    By Sami4u in forum Building
    Replies: 9
    Last Post: 27 September, 2009, 07:43 AM
  3. Robots.txt?
    By dmi in forum Managing
    Replies: 15
    Last Post: 9 September, 2009, 17:15 PM
  4. Block Robots and Web Downloaders with robots.txt
    By Will.Spencer in forum Managing
    Replies: 12
    Last Post: 6 June, 2009, 15:40 PM
  5. What is robots.txt file?
    By ltimranjaved in forum Managing
    Replies: 1
    Last Post: 26 May, 2009, 12:27 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •