Thread: Google is indexing categories...

I have recently started a blog and I am using robots.txt file to restrict Google bots of some areas but that's not working I think. This is my Robots.txt file

User-agent: *
Disallow: /wp-content/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-
Disallow: /feed/
Disallow: /trackback/
Disallow: /cgi-bin/

User-agent: Googlebot
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /feed/
Disallow: /archives/
Disallow: /index.php
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/
Disallow: /category/
Disallow: /*?
Disallow: */comment-page
Allow: /wp-content/uploads/

User-agent: Googlebot-Image
Allow: /*

Sitemap: http://www.techreaders.com/sitemap.xml.gz

Google seems to index my categories as well as plugins...

site:www.techreaders.com - Google Search

Check last 4 results. Any help will be highly appreciated.

Thank you

According the Google support pages (emphasis added) :

A robots.txt file restricts access to your site by search engine robots that crawl the web. (Note, however, that while Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web.) To use a robots.txt file, you'll need to have root access to your server.

Which could explain why your category urls are being indexed, but with out any content description.

Can you add a noindex metatag to these pages? But then that brings up the question, "If google supposedly won't crawl them because of the robots.txt file, how will they see the noindex tag?" hmm.

Considering your on WP, install the All-In-One SEO plugin, there one of the options in the 'admin options' is 'use noindex' for categories and tags. You may 'check' it.

Originally Posted by Snak3

Considering your on WP, install the All-In-One SEO plugin, there one of the options in the 'admin options' is 'use noindex' for categories and tags. You may 'check' it.

I agree, If you are on wordpress then this is the best option to keep your categories from being indexed, not sure why you want to do that though.

Originally Posted by Seattle SEO

I agree, If you are on wordpress then this is the best option to keep your categories from being indexed, not sure why you want to do that though.

It is a fairly common practice to block categories in WordPress to avoid potential duplicate content issues. all-in-one-seo-pack can be set to add noindex,follow robots meta tags to the category pages. I strongly recommend it. There isn't any value in having search engines index category pages. Keep then focused on your articles. You want them to follow the links on the category pages, but skip indexing the category pages.

I don't know why your plugin directories are being indexed. There should not be any visible links to the plugin directories in your client-side code. You might want to scan your client-side code (using View, Source) to see if links to the directories are showing up in the code.

BTW, your server is displaying the contents of the directories. It should not be doing that. It is a security issue.

You can find a fix here:

Prevent Directory Listing - Web Site Security Tutorial | Tech-Evangelist

Google will probably stop indexing the plugin directories if they start generating a status code 403 when an index file does not exist, which is what they should be doing.

To use a robots.txt file, you'll need to have root access to your server.

If you got this from a Google page, the writer doesn't know very much about web servers. Having root access to a server is something entirely different from placing a file in a site's root directory. Every site owner has FTP access to the root directory, but most hosting companies do not allow root access to the server.

Use the All In One SEO or Robots META Plugin for Wordpress and adjust the settings so that those pages are not indexed. Don't get rid of your robots.txt file, though - it's best to use both.

Then login to the Google Webmaster's panel. Click Crawler Access and then Remove URLs. Enter all the indexed category URLs and Google will remove them.

Originally Posted by TopDogger

...
If you got this from a Google page, the writer doesn't know very much about web servers. Having root access to a server is something entirely different from placing a file in a site's root directory. Every site owner has FTP access to the root directory, but most hosting companies do not allow root access to the server.

I did (Removing my own content from Google) and yes I know and agree.

Thank you guys for the help. I dont see my files shown up....:s It got fixed automatically I guess

My tags are set 'follow'. May be I should try to set them to 'no follow' but I am confused. I had a blog a while ago and the tags were indexing and I got most of the traffic from tags....