Page 2 of 2 FirstFirst 12
Results 11 to 13 of 13

Thread: Block Robots and Web Downloaders with robots.txt

  1. #11
    Aquarezz's Avatar
    Aquarezz is offline Master Net Builder
    Join Date
    Dec 2008
    Location
    Belgium
    Posts
    3,853
    Blog Entries
    4
    Thanks
    683
    Thanked 591 Times in 463 Posts
    Quote Originally Posted by 1901gt View Post
    Actually, what is the purpose of this robots.txt?
    There are website downloaders on the web, they go through the hole website and download every signle page. A normal internet user visiting your website might use a lot of bandwidth already, but if a bot like this does it it uses the bandwidth all over your site -- That's why it's better to block it, people should visit your site online instead of downloading it and viewing it on their PC. (You won't get any adsense revenue from that neither, so it's a lose-lose situation)

    - Aquarezz
    |Nico Lawsons

  2. Thanked by:

    1901gt (6 June, 2009)

  3. #12
    1901gt is offline Web Designer
    Join Date
    Jan 2009
    Location
    Singapore
    Posts
    235
    Blog Entries
    2
    Thanks
    35
    Thanked 32 Times in 24 Posts
    And how exactly do I block these bots? Robots.txt or .htaccess? Whats the difference between them both?

  4. #13
    Aquarezz's Avatar
    Aquarezz is offline Master Net Builder
    Join Date
    Dec 2008
    Location
    Belgium
    Posts
    3,853
    Blog Entries
    4
    Thanks
    683
    Thanked 591 Times in 463 Posts
    Quote Originally Posted by 1901gt View Post
    And how exactly do I block these bots? Robots.txt or .htaccess? Whats the difference between them both?
    Best is .htaccess, as these bots can more easy bypass a robots.txt file than that they can bypass a .htaccess file

    - Aquarezz
    |Nico Lawsons

  5. Thanked by:

    1901gt (6 June, 2009)

Page 2 of 2 FirstFirst 12

Similar Threads

  1. The EPFL mini-robots
    By kiki in forum General Chat
    Replies: 1
    Last Post: 2 June, 2010, 03:24 AM
  2. Help with Robots.txt
    By 5starpix in forum Building
    Replies: 4
    Last Post: 11 February, 2010, 03:27 AM
  3. robots.txt help
    By Sami4u in forum Building
    Replies: 9
    Last Post: 27 September, 2009, 08:43 AM
  4. Robots.txt?
    By dmi in forum Managing
    Replies: 15
    Last Post: 9 September, 2009, 18:15 PM
  5. What is robots.txt file?
    By ltimranjaved in forum Managing
    Replies: 1
    Last Post: 26 May, 2009, 13:27 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •