Will.Spencer

I have found Bot Trap to be extremely effective in blocking the sneakiest of the web scraper bots.

Bot Trap works by placing a hidden link on your homepage. That link can only be seen in the source code to the page. Ergo, the only things that should see and follow that link are web robots. But, that link is Disallowed in robots.txt, so polite robots will never try to follow that link.

Robots that do follow that link get automatically added with Deny statements into your sites .htaccess file.

Sometimes legitimate web robots get out of sync, so to make the script able to run unattended, I recommend that you whitelist those in your .htaccess file.

Here's my current whitelist:
This blocks people stealing your content to place on MFA sites and it blocks people downloading your entire websites for offline reading. I constantly see people trying to download my fifty-thousand page web sites. It's a complete waste of bandwidth.

Bot Trap is friendly though. Users will see a message telling them that they are blocked and they only have to enter the word "access" into a form to be automatically unblocked.

Every blocking or unblocking action generates an email to the site admin.

It's really a beautiful script.

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

freexs.org

I don't get this, you allow some search bots and you block all others, how do you do that with real visitors? they have an IP address too

Will.Spencer

Only search bots which access the hidden bot-trap file get blocked.

Real visitors get blocked if they access that hidden file also, but they can unblock themselves just by typing "access" into the form.

Since search bots tend to be stupid, I specifically whitelist (allow) the important ones. This means that they can't get themselves blocked, even if they do access the hidden bot-trap file.

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

DomainMagnate

Why not just allow the known se bots and disallow the rest?

Will.Spencer

The normal way to do that would be to block them in robots.txt, but the bad guys just ignore robots.txt.

To deny them in .htaccess, you need to know their IP addresses, and the really bad guys change IP's frequently.

But it's not just bots. Web downloaders like wget, HTTrack, and WebCopier each up huge amounts of bandwidth and provide almost no value to the web site owner.

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

firetown

Someone on Sphinn asks:

Who other than the major search engines would you add to your white list? Will you be hurting your rank if you keep the scrapers from grabbing your content?

__________________
Mortgage information and debt consolidation
Villas in Bali, Indonesia- Firetown - do Secret Societies Exist?

Mr.Bill

I use robot trap on a couple sites. However coming the beginning of next year might not be as important anymore.


Scrapers are not going to fair well
webpronews.com/topnews/2008/11/17/seo-about-to-get-turned-on-its-ear

__________________

Will.Spencer

I don't have much faith in personalized search, but that's a whole other thread.

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

EricBlackwell

I agree with Will's assessment of personalized search...but back onto the main topic...

Up until now, I have not worried too much about scrapers, but just noticed one of my main real estate sites getting scraped for listings by a major national third party site... time to take action, I guess....

Thanks for the how to on this.

Eric

__________________
Eric Blackwell
SEO, Technologist, Web Entrepreneur

My Search Engine Marketing Blog
The Eric Blackwell Personal Blog
My latest real estate project.

Too many other projects to mention...hehe

TopDogger

Will, does your white list cover all of the Google spiders?

I've seen some lengthy lists of GoogleBot IP addresses.