Page 5 of 5 FirstFirst ... 345
Results 41 to 44 of 44

Thread: Block Web Content Scrapers and Downloaders

  1. #41
    You will find hundreds of new spiders with bot-trap.
    Last edited by TopDogger; 27 May, 2011 at 13:31 PM.
    "Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote." -- Benjamin Franklin


  2. #42
    ## this line won't be read as a command (or at all) so it won't create an error. But, if you get a hash mark in front of some code - then...

    Anyone know where I can find an up to date comprehensive list of SE user agents? I need the names for white list, not IP numbers

  3. #43
    Just do a search for 'search engine user agents'. Lots of sites come up.

    List of User-Agents (Spiders, Robots, Browser)

    It seems like recently an entire new group of spiders has been unleashed. I'm currently trapping dozens of new spiders every day.
    Last edited by TopDogger; 30 May, 2011 at 12:27 PM.
    "Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote." -- Benjamin Franklin


  4. #44
    If you like Blekko, add these IP ranges to your Allow list.

    Allow from 38.99.96.0/24
    Allow from 64.13.159.0/24
    Allow from 199.87.248.0/21
    "Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote." -- Benjamin Franklin


Page 5 of 5 FirstFirst ... 345

Similar Threads

  1. Obfuscate Proxy Content to make harder to Block
    By tibbie in forum Web Proxies
    Replies: 4
    Last Post: 2 May, 2011, 08:34 AM
  2. New Content writer on the block!
    By AjiContent in forum Introduction Forum
    Replies: 0
    Last Post: 28 February, 2011, 09:19 AM
  3. Google on Content Scrapers
    By Kovich in forum Managing
    Replies: 18
    Last Post: 14 May, 2010, 06:58 AM
  4. Block Robots and Web Downloaders with robots.txt
    By Will.Spencer in forum Managing
    Replies: 12
    Last Post: 6 June, 2009, 15:40 PM
  5. How to Profit from Content Scrapers?
    By Shenron in forum Promoting
    Replies: 4
    Last Post: 12 March, 2009, 18:58 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •