You will find hundreds of new spiders with bot-trap.
Printable View
You will find hundreds of new spiders with bot-trap.
## this line won't be read as a command (or at all) so it won't create an error. But, if you get a hash mark in front of some code - then...
Anyone know where I can find an up to date comprehensive list of SE user agents? I need the names for white list, not IP numbers
Just do a search for 'search engine user agents'. Lots of sites come up.
List of User-Agents (Spiders, Robots, Browser)
It seems like recently an entire new group of spiders has been unleashed. I'm currently trapping dozens of new spiders every day.
If you like Blekko, add these IP ranges to your Allow list.
Allow from 38.99.96.0/24
Allow from 64.13.159.0/24
Allow from 199.87.248.0/21