Results 1 to 5 of 5

Thread: Help with Robots.txt

  1. #1
    5starpix's Avatar
    5starpix is offline Senior Net Builder
    Join Date
    Dec 2008
    Location
    Montreal, Quebec
    Posts
    1,710
    Blog Entries
    9
    Thanks
    158
    Thanked 227 Times in 164 Posts

    Question Help with Robots.txt

    Hello

    I used the Google Webmaster tool to create a robots.txt file.

    I got this;

    User-agent: *
    Allow: /
    Now, if I wanted the bots to not index/crawl my images folder, what would I add?

    Also, is my current robots.txt file alright?

  2. #2
    TopDogger's Avatar
    TopDogger is offline Über Hund
    Join Date
    Jan 2009
    Location
    Hellfire, AZ
    Posts
    3,079
    Thanks
    347
    Thanked 914 Times in 698 Posts
    The problem with Google Webmaster Tools is that they create and validate a robots.txt file that Google recognizes, but it does not adhere to the standard. The Allow directive is not part of the robots.txt standard. Other spiders may recognize it, but you will not find the Allow directive to be part of the standard specifications.

    If you just want to block the images folder, use the following:

    User-agent: *
    Disallow: /images/
    "Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote." -- Benjamin Franklin


  3. Thanked by:

    5starpix (21 December, 2009)

  4. #3
    Jammyjoker's Avatar
    Jammyjoker is offline Jammy Joker
    Join Date
    Dec 2009
    Location
    India
    Posts
    58
    Thanks
    0
    Thanked 3 Times in 3 Posts
    The Web Robots Pages - Hopes this may help you. best of luck.

  5. Thanked by:

    Snak3 (21 December, 2009)

  6. #4
    5starpix's Avatar
    5starpix is offline Senior Net Builder
    Join Date
    Dec 2008
    Location
    Montreal, Quebec
    Posts
    1,710
    Blog Entries
    9
    Thanks
    158
    Thanked 227 Times in 164 Posts
    Quote Originally Posted by TopDogger View Post
    The problem with Google Webmaster Tools is that they create and validate a robots.txt file that Google recognizes, but it does not adhere to the standard. The Allow directive is not part of the robots.txt standard. Other spiders may recognize it, but you will not find the Allow directive to be part of the standard specifications.

    If you just want to block the images folder, use the following:

    Thanks. I have added that to my file

  7. #5
    5starpix's Avatar
    5starpix is offline Senior Net Builder
    Join Date
    Dec 2008
    Location
    Montreal, Quebec
    Posts
    1,710
    Blog Entries
    9
    Thanks
    158
    Thanked 227 Times in 164 Posts

    Smile

    Quote Originally Posted by cuinee View Post
    You use the following syntax:

    Code:
    User-agent: *
    Disallow: /images/
    That is exactly what TopDogger wrote..


Similar Threads

  1. The EPFL mini-robots
    By kiki in forum General Chat
    Replies: 1
    Last Post: 2 June, 2010, 02:24 AM
  2. robots.txt help
    By Sami4u in forum Building
    Replies: 9
    Last Post: 27 September, 2009, 07:43 AM
  3. Robots.txt?
    By dmi in forum Managing
    Replies: 15
    Last Post: 9 September, 2009, 17:15 PM
  4. Block Robots and Web Downloaders with robots.txt
    By Will.Spencer in forum Managing
    Replies: 12
    Last Post: 6 June, 2009, 15:40 PM
  5. What is robots.txt file?
    By ltimranjaved in forum Managing
    Replies: 1
    Last Post: 26 May, 2009, 12:27 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •