LinkBack URL
About LinkBacksYou will find hundreds of new spiders with bot-trap.
Über Hund
You will find hundreds of new spiders with bot-trap.
Last edited by TopDogger; 27 May, 2011 at 13:31 PM.
You can have it fast, good or cheap. Pick any two.
siteowner
## this line won't be read as a command (or at all) so it won't create an error. But, if you get a hash mark in front of some code - then...
Anyone know where I can find an up to date comprehensive list of SE user agents? I need the names for white list, not IP numbers
Über Hund
Just do a search for 'search engine user agents'. Lots of sites come up.
List of User-Agents (Spiders, Robots, Browser)
It seems like recently an entire new group of spiders has been unleashed. I'm currently trapping dozens of new spiders every day.
Last edited by TopDogger; 30 May, 2011 at 12:27 PM.
You can have it fast, good or cheap. Pick any two.
Über Hund
If you like Blekko, add these IP ranges to your Allow list.
Allow from 38.99.96.0/24
Allow from 64.13.159.0/24
Allow from 199.87.248.0/21
You can have it fast, good or cheap. Pick any two.
Will.Spencer (26 March, 2012)
Posting Permissions
Reply With Quote
Bookmarks