Block Robots and Web Downloaders with robots.txt

Printable View

6 June, 2009, 12:02 PM
Aquarezz

Quote:

Originally Posted by 1901gt

Actually, what is the purpose of this robots.txt?

There are website downloaders on the web, they go through the hole website and download every signle page. A normal internet user visiting your website might use a lot of bandwidth already, but if a bot like this does it it uses the bandwidth all over your site -- That's why it's better to block it, people should visit your site online instead of downloading it and viewing it on their PC. (You won't get any adsense revenue from that neither, so it's a lose-lose situation)

- Aquarezz
6 June, 2009, 13:23 PM
1901gt

And how exactly do I block these bots? Robots.txt or .htaccess? Whats the difference between them both?
6 June, 2009, 15:40 PM
Aquarezz

Quote:

Originally Posted by 1901gt

And how exactly do I block these bots? Robots.txt or .htaccess? Whats the difference between them both?

Best is .htaccess, as these bots can more easy bypass a robots.txt file than that they can bypass a .htaccess file ;)

- Aquarezz