Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: SimilarPages

  1. #1
    Will.Spencer's Avatar
    Will.Spencer is offline Retired
    Join Date
    Dec 2008
    Posts
    5,033
    Blog Entries
    1
    Thanks
    1,010
    Thanked 2,329 Times in 1,259 Posts

    SimilarPages

    A web robot claiming to be from SimilarPages has been crawling my sites, while completely ignoring robots.txt.

    It's difficult to tell for certain that this robot really is from SimilarPages.com, because the crawling is coming from Amazon's AWS cloud computing system.

    The robot visits look like this:
    2009-02-25 (Wed) 02:20:40 address is 67.202.52.58, hostname is ec2-67-202-52-58.compute-1.amazonaws.com, agent is SimilarPages/Nutch-1.0-dev (SimilarPages Nutch Crawler; http://www.similarpages.com; info at similarpages dot com)
    If this bot really is from SimilarPages.com, I want to tell them that they aren't going to beat Google by building a search engine from an open source project (Nutch) and hosted at Amazon Web Services.

    If this is just another blackhat crawler hosted at Amazon, I want to tell Amazon that their customers are hurting their brand value.

    I think that I will block Amazon Web Services's IP ranges on our external firewall.
    Submit Your Webmaster Related Sites to the NB Directory
    I swear, by my life and my love of it, that I will never live for the sake of another man, nor ask another man to live for mine.

  2. Thanked by:

    zaphod (7 April, 2009)

  3. #2
    TopDogger's Avatar
    TopDogger is offline Über Hund
    Join Date
    Jan 2009
    Location
    Hellfire, AZ
    Posts
    3,076
    Thanks
    347
    Thanked 913 Times in 697 Posts
    That one looks very suspicious. Even is it is from SimilarPages.com, what would be the advantage with letting it crawl your sites? Does anyone actually use SimilarPages.com?

    I would block it.

  4. #3
    Desan is offline Moderator
    Join Date
    Dec 2008
    Location
    Finland
    Posts
    469
    Blog Entries
    2
    Thanks
    101
    Thanked 52 Times in 35 Posts
    I woul block it as well. Even if it is from similarpages.com, it wouldn't be big loss.

  5. #4
    Aquarezz's Avatar
    Aquarezz is offline Master Net Builder
    Join Date
    Dec 2008
    Location
    Belgium
    Posts
    3,852
    Blog Entries
    4
    Thanks
    683
    Thanked 591 Times in 463 Posts
    They're trying to build a new Copyscape website maybe Maybe they will only do it paid and will only get 100 people buying it monthly, but if it will cost 10$ they will have an extra 1000$

    Just a quick stupid theory of mine.. but their idea is probably based on Copyscape (As their name is similarpages)

    Well block them, that's all I can say


    Greetz
    |Nico Lawsons

  6. #5
    Shawn's Avatar
    Shawn is offline Net Builder
    Join Date
    Dec 2008
    Location
    Dominican Republic
    Posts
    165
    Thanks
    10
    Thanked 25 Times in 22 Posts
    There website is just a logo and here is some whois data:
    Similarpages.com - Similar Pages

    I would say either someone is trying to build something new or just harvesting for their own purpose.

  7. #6
    TopDogger's Avatar
    TopDogger is offline Über Hund
    Join Date
    Jan 2009
    Location
    Hellfire, AZ
    Posts
    3,076
    Thanks
    347
    Thanked 913 Times in 697 Posts
    It it cannot be a serious site when it is hosted at Yahoo and sneaking in through Amazon's AWS Cloud Computing.

    I think they are trying to create similar pages, AKA spamola.

  8. #7
    stickycarrots's Avatar
    stickycarrots is offline Experienced Net Builder
    Join Date
    Dec 2008
    Location
    QuickInbox.com
    Posts
    753
    Blog Entries
    6
    Thanks
    18
    Thanked 86 Times in 59 Posts
    That bot has been roaming around my blog as well. I like him.

  9. #8
    Shawn's Avatar
    Shawn is offline Net Builder
    Join Date
    Dec 2008
    Location
    Dominican Republic
    Posts
    165
    Thanks
    10
    Thanked 25 Times in 22 Posts
    Quote Originally Posted by stickycarrots View Post
    That bot has been roaming around my blog as well. I like him.
    Is he your pet?

  10. #9
    stickycarrots's Avatar
    stickycarrots is offline Experienced Net Builder
    Join Date
    Dec 2008
    Location
    QuickInbox.com
    Posts
    753
    Blog Entries
    6
    Thanks
    18
    Thanked 86 Times in 59 Posts
    Quote Originally Posted by Shawn View Post
    Is he your pet?
    I have named him Spot.

  11. #10
    Will.Spencer's Avatar
    Will.Spencer is offline Retired
    Join Date
    Dec 2008
    Posts
    5,033
    Blog Entries
    1
    Thanks
    1,010
    Thanked 2,329 Times in 1,259 Posts
    Quote Originally Posted by stickycarrots View Post
    I have named him Spot.
    Out, damn'd spot!
    Submit Your Webmaster Related Sites to the NB Directory
    I swear, by my life and my love of it, that I will never live for the sake of another man, nor ask another man to live for mine.

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •