Thread: Site map?

Is netbuilders submitting a sitemap to google webmaster tools?

We're not. The only real purpose of XML sitemaps is to work around crawlability problems, and we have no problems with crawlability.

Do you still get to see all the cool stats in the GWT even if you don't have a sitemap?

Originally Posted by Will.Spencer

We're not. The only real purpose of XML sitemaps is to work around crawlability problems, and we have no problems with crawlability.

Actually, Google is only showing 1,600 pages indexed from NetBuilders in my results. We have nearly 5,000 threads. Why aren't those others indexed? I've found pinging Google with your sitemap helps get indexed deeper. Why not try it?

Originally Posted by Will.Spencer

We're not. The only real purpose of XML sitemaps is to work around crawlability problems, and we have no problems with crawlability.

Will, when I checked to see how many pages netbuilders has indexed, I was suprised. Google only has about 1,670 pages indexed for netbuilders.

For the amount of threads and post you have, that is low. And I am not the only one seeing this.

Originally Posted by stickycarrots

Actually, Google is only showing 1,600 pages indexed from NetBuilders in my results. We have nearly 5,000 threads. Why aren't those others indexed? I've found pinging Google with your sitemap helps get indexed deeper. Why not try it?

That's an easy answer -- we don't allow Googlebot to index all of our threads.

From our robots.txt:

User-agent: *
Disallow: /politics-religion/
Disallow: /general-chat/
Disallow: /entertainment/

My experience with some other webmaster forums is that if you allow these threads to get indexed, you will attract a lot of non-webmasters who are interested only in trolling or spamming. This eventually prevents real community members from engaging in genuine conversation.

We have the other forums crawlable, because we want to attract people interested in those topics.

10-4 Will, I fully understand why you added those forums to your robots.txt. And it also explains why google has not indexed a lot of your threads and post.

Originally Posted by Will.Spencer

That's an easy answer -- we don't allow Googlebot to index all of our threads.

From our robots.txt:

User-agent: *
Disallow: /politics-religion/
Disallow: /general-chat/
Disallow: /entertainment/

My experience with some other webmaster forums is that if you allow these threads to get indexed, you will attract a lot of non-webmasters who are interested only in trolling or spamming. This eventually prevents real community members from engaging in genuine conversation.

We have the other forums crawlable, because we want to attract people interested in those topics.

Hm, good thinking. I never thought of doing this.

Have you experimented with this on and off (perhaps on another website)? If so, what were the results like?

I'd imagine there could only be one result. That the threads in those forum sections aren't being indexed and the amount of targeted traffic netbuilders receives is higher

Originally Posted by Nick

Have you experimented with this on and off (perhaps on another website)? If so, what were the results like?

Not yet, this is the first time I thought of it.