Thread: Block Web Content Scrapers and Downloaders

I received a notification from bot-trap that Microsoft is using a new crawling address for Bingbot:

A bad robot hit /bot-trap/index.php 2010-12-24 (Fri) 18:03:52
address is 207.46.12.240, hostname is msnbot-207-46-12-240.search.msn.com, agent is Mozilla/5.0 (compatible; bingbot/2.0; +Bing Webmaster Center)

207.46.12.240 is in the network range 207.46.0.0 - 207.46.255.255, which is 207.46.0.0/16 in CIDR notation.

If you use bot-trap, you'll want to add a line like this to your .htaccess file:

Allow from 207.46

Does it matter whether I use the CIDR notation or 206.46?

Does one work any better than the other?

I already added the CIDR notation.

I have noticed that sometimes bad spiders slip past Deny statements that use CIDR and end up getting caught by bot-trap. If the CIDR Deny statement was working, they should not have the opportunity to get caught by bot-trap.

bot-trap snagged 775 bad spiders in my sites in the last two months.

I wasn't sure that CIDR worked in .htaccess, but I found How to use a CIDR netmask to block an IP address range in .htaccess so I guess it does.

CIDR does appear to work about 99% of the time. I used it to block the multiple ranges of Yandex bots that were ignoring the robots.txt file and killing my bandwidth. However, every once in a while a Yandex spider appears to slip through.

Microsoft is making more changes:

Code:

A bad robot hit /bot-trap/index.php 2010-12-26 (Sun) 07:23:51 
address is 157.55.116.57, hostname is msnbot-157-55-116-57.search.msn.com, agent is msnbot/2.0b (+http://search.msn.com/msnbot.htm).

WHOIS gives us three CIDR ranges from that IP address:

Allow from 157.54.0.0/15
Allow from 157.56.0.0/14
Allow from 157.60.0.0/16

how to block browser scrapper ? cause they using dynamic IP, basically nothing much i can do other then human monitor right ??

they creating 10 mb inbound traffic and high latency almost cause my server stop ....

Originally Posted by SonnyCooL

how to block browser scrapper ? cause they using dynamic IP, basically nothing much i can do other then human monitor right ??

bot-trap finds the scrapers at each new IP. Give it a try. Seriously.

Originally Posted by Will.Spencer

bot-trap finds the scrapers at each new IP. Give it a try. Seriously.

but those dynamic IP will reuse again by another human sooner or later

ok saw the captcha for human thing will try it out later with my min knowledge

Originally Posted by SonnyCooL

but those dynamic IP will reuse again by another human sooner or later

If you're worried, you can clean the banned IP addresses out of .htaccess once a month or so.

this looks like an interesting piece of software will try on my website.

in the mean time i generally block spam bots and leachers using useragent stings through ht access.

htaccess based spamBot and Leacher Blocking Code | Anant Shrivastava : Techno Enthusiast