Results 1 to 5 of 5

Thread: Need a Website Crawler

  1. #1
    bhartzer is offline Net Builder
    Join Date
    Dec 2008
    Posts
    502
    Thanks
    53
    Thanked 135 Times in 101 Posts

    Need a Website Crawler

    I'm looking for a tool (either a script or something I can run locally on my laptop) that will crawl an entire website and give me this info on all pages on the site:

    url
    title tag
    meta description tag
    meta keywords tag

    Do you know of a script or website crawler program that will do that?
    Need links? Try AuthorLinks where you can buy or sell links based on Authorship and Klout score. Check out my blog or like me on Facebook.

  2. #2
    SonnyCooL's Avatar
    SonnyCooL is offline HeeHa
    Join Date
    Jan 2010
    Location
    Melb/Malaysia
    Posts
    920
    Thanks
    250
    Thanked 92 Times in 78 Posts
    custom will be the best ..... can save under any format

  3. #3
    Mike-XS's Avatar
    Mike-XS is offline XeroAgent
    Join Date
    Sep 2009
    Location
    OZ
    Posts
    209
    Thanks
    30
    Thanked 109 Times in 71 Posts
    Hi,

    I don't recall seeing anything that does exactly what you need, but maybe you could use this script below to build a list of all the urls on a site.

    PHP Examples: Extract all URLS from a webpage

    and then feed that list into a script like this.

    getUrlInfo Demo
    Bot that retrieves url meta data and other infos - Barattalo
    Mini Bots PHP Class - Barattalo

    There's a ton of similar scripts which could be useful, I'll keep an eye out for something better.

  4. Thanked by:

    Sami4u (29 December, 2010), TopDogger (30 December, 2010)

  5. #4
    bhartzer is offline Net Builder
    Join Date
    Dec 2008
    Posts
    502
    Thanks
    53
    Thanked 135 Times in 101 Posts
    Great, thanks Mike-XS, I'll try those scripts.

    I have OptiSpider, which seems to do a great job, but it doesn't grab the meta description tags.
    Need links? Try AuthorLinks where you can buy or sell links based on Authorship and Klout score. Check out my blog or like me on Facebook.

  6. #5
    Will.Spencer's Avatar
    Will.Spencer is offline Retired
    Join Date
    Dec 2008
    Posts
    5,033
    Blog Entries
    1
    Thanks
    1,010
    Thanked 2,327 Times in 1,258 Posts
    The Screaming Frog SEO Spider grabs the meta description tag.

    If the site is Wordpress, BlogLinkspider is an incredible shortcut.
    Submit Your Webmaster Related Sites to the NB Directory
    I swear, by my life and my love of it, that I will never live for the sake of another man, nor ask another man to live for mine.

  7. Thanked by:

    bhartzer (17 January, 2011)

Similar Threads

  1. Search Engine Crawler and Spider
    By joy1986joy in forum Promoting
    Replies: 4
    Last Post: 20 July, 2010, 13:20 PM
  2. Replies: 7
    Last Post: 24 March, 2010, 17:30 PM
  3. Replies: 7
    Last Post: 29 January, 2010, 10:21 AM
  4. Bing's crawler still slow?
    By tmongy in forum Search Engine News
    Replies: 5
    Last Post: 5 September, 2009, 10:24 AM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •