Printable View
I'm looking for a tool (either a script or something I can run locally on my laptop) that will crawl an entire website and give me this info on all pages on the site:
url
title tag
meta description tag
meta keywords tag
Do you know of a script or website crawler program that will do that?
custom will be the best ..... can save under any format :)
Hi,
I don't recall seeing anything that does exactly what you need, but maybe you could use this script below to build a list of all the urls on a site.
PHP Examples: Extract all URLS from a webpage
and then feed that list into a script like this.
getUrlInfo Demo
Bot that retrieves url meta data and other infos - Barattalo
Mini Bots PHP Class - Barattalo
There's a ton of similar scripts which could be useful, I'll keep an eye out for something better.
Great, thanks Mike-XS, I'll try those scripts.
I have OptiSpider, which seems to do a great job, but it doesn't grab the meta description tags.
The Screaming Frog SEO Spider grabs the meta description tag.
If the site is Wordpress, BlogLinkspider is an incredible shortcut.