Page 2 of 2 FirstFirst 12
Results 11 to 13 of 13

Thread: Block Robots and Web Downloaders with robots.txt

  1. #11
    Quote Originally Posted by 1901gt View Post
    Actually, what is the purpose of this robots.txt?
    There are website downloaders on the web, they go through the hole website and download every signle page. A normal internet user visiting your website might use a lot of bandwidth already, but if a bot like this does it it uses the bandwidth all over your site -- That's why it's better to block it, people should visit your site online instead of downloading it and viewing it on their PC. (You won't get any adsense revenue from that neither, so it's a lose-lose situation)

    - Aquarezz
    |Nico Lawsons

  2. #12
    And how exactly do I block these bots? Robots.txt or .htaccess? Whats the difference between them both?

  3. #13
    Quote Originally Posted by 1901gt View Post
    And how exactly do I block these bots? Robots.txt or .htaccess? Whats the difference between them both?
    Best is .htaccess, as these bots can more easy bypass a robots.txt file than that they can bypass a .htaccess file

    - Aquarezz
    |Nico Lawsons

Page 2 of 2 FirstFirst 12

Similar Threads

  1. The EPFL mini-robots
    By kiki in forum General Chat
    Replies: 1
    Last Post: 2 June, 2010, 02:24 AM
  2. Help with Robots.txt
    By 5starpix in forum Building
    Replies: 4
    Last Post: 11 February, 2010, 03:27 AM
  3. robots.txt help
    By Sami4u in forum Building
    Replies: 9
    Last Post: 27 September, 2009, 07:43 AM
  4. Robots.txt?
    By dmi in forum Managing
    Replies: 15
    Last Post: 9 September, 2009, 17:15 PM
  5. What is robots.txt file?
    By ltimranjaved in forum Managing
    Replies: 1
    Last Post: 26 May, 2009, 12:27 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •