Results 1 to 5 of 5

Thread: Need a Website Crawler

Hybrid View

  1. Need a Website Crawler

    I'm looking for a tool (either a script or something I can run locally on my laptop) that will crawl an entire website and give me this info on all pages on the site:

    url
    title tag
    meta description tag
    meta keywords tag

    Do you know of a script or website crawler program that will do that?
    Need links? Try AuthorLinks where you can buy or sell links based on Authorship and Klout score. Check out my blog or like me on Facebook.

  2. #2
    custom will be the best ..... can save under any format

  3. #3
    Hi,

    I don't recall seeing anything that does exactly what you need, but maybe you could use this script below to build a list of all the urls on a site.

    PHP Examples: Extract all URLS from a webpage

    and then feed that list into a script like this.

    getUrlInfo Demo
    Bot that retrieves url meta data and other infos - Barattalo
    Mini Bots PHP Class - Barattalo

    There's a ton of similar scripts which could be useful, I'll keep an eye out for something better.

  4. Great, thanks Mike-XS, I'll try those scripts.

    I have OptiSpider, which seems to do a great job, but it doesn't grab the meta description tags.
    Need links? Try AuthorLinks where you can buy or sell links based on Authorship and Klout score. Check out my blog or like me on Facebook.

  5. The Screaming Frog SEO Spider grabs the meta description tag.

    If the site is Wordpress, BlogLinkspider is an incredible shortcut.
    Submit Your Webmaster Related Sites to the NB Directory
    I swear, by my life and my love of it, that I will never live for the sake of another man, nor ask another man to live for mine.

Similar Threads

  1. Search Engine Crawler and Spider
    By joy1986joy in forum Promoting
    Replies: 4
    Last Post: 20 July, 2010, 13:20 PM
  2. Replies: 7
    Last Post: 24 March, 2010, 17:30 PM
  3. Replies: 7
    Last Post: 29 January, 2010, 11:21 AM
  4. Bing's crawler still slow?
    By tmongy in forum Search Engine News
    Replies: 5
    Last Post: 5 September, 2009, 10:24 AM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •