Results 1 to 8 of 8

Thread: Google is indexing categories...

  1. #1
    F-GD's Avatar
    F-GD is offline Lost Member
    Join Date
    Jan 2009
    Posts
    159
    Thanks
    65
    Thanked 24 Times in 17 Posts

    Google is indexing categories...

    I have recently started a blog and I am using robots.txt file to restrict Google bots of some areas but that's not working I think. This is my Robots.txt file

    User-agent: *
    Disallow: /wp-content/
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Disallow: /wp-
    Disallow: /feed/
    Disallow: /trackback/
    Disallow: /cgi-bin/

    User-agent: Googlebot
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /feed/
    Disallow: /archives/
    Disallow: /index.php
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: */feed/
    Disallow: */trackback/
    Disallow: /category/
    Disallow: /*?
    Disallow: */comment-page
    Allow: /wp-content/uploads/

    User-agent: Googlebot-Image
    Allow: /*

    Sitemap: http://www.techreaders.com/sitemap.xml.gz
    Google seems to index my categories as well as plugins...

    Check last 4 results. Any help will be highly appreciated.

    Thank you

  2. #2
    dtkguy's Avatar
    dtkguy is offline Still Learnin'
    Join Date
    Dec 2008
    Posts
    391
    Thanks
    7
    Thanked 164 Times in 108 Posts
    According the Google support pages (emphasis added) :

    A robots.txt file restricts access to your site by search engine robots that crawl the web. (Note, however, that while Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web.) To use a robots.txt file, you'll need to have root access to your server.
    Which could explain why your category urls are being indexed, but with out any content description.

    Can you add a noindex metatag to these pages? But then that brings up the question, "If google supposedly won't crawl them because of the robots.txt file, how will they see the noindex tag?" hmm.

  3. Thanked by:

    F-GD (25 August, 2009)

  4. #3
    Snak3's Avatar
    Snak3 is offline Moderator
    Join Date
    Jul 2009
    Location
    Undisclosed Location
    Posts
    629
    Thanks
    155
    Thanked 190 Times in 121 Posts
    Considering your on WP, install the All-In-One SEO plugin, there one of the options in the 'admin options' is 'use noindex' for categories and tags. You may 'check' it.

  5. #4
    Seattle SEO is offline Unknown Net Builder
    Join Date
    Aug 2009
    Posts
    11
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Snak3 View Post
    Considering your on WP, install the All-In-One SEO plugin, there one of the options in the 'admin options' is 'use noindex' for categories and tags. You may 'check' it.
    I agree, If you are on wordpress then this is the best option to keep your categories from being indexed, not sure why you want to do that though.

  6. #5
    TopDogger's Avatar
    TopDogger is offline Über Hund
    Join Date
    Jan 2009
    Location
    Hellfire, AZ
    Posts
    2,946
    Thanks
    341
    Thanked 883 Times in 671 Posts
    Quote Originally Posted by Seattle SEO View Post
    I agree, If you are on wordpress then this is the best option to keep your categories from being indexed, not sure why you want to do that though.
    It is a fairly common practice to block categories in WordPress to avoid potential duplicate content issues. all-in-one-seo-pack can be set to add noindex,follow robots meta tags to the category pages. I strongly recommend it. There isn't any value in having search engines index category pages. Keep then focused on your articles. You want them to follow the links on the category pages, but skip indexing the category pages.

    I don't know why your plugin directories are being indexed. There should not be any visible links to the plugin directories in your client-side code. You might want to scan your client-side code (using View, Source) to see if links to the directories are showing up in the code.

    BTW, your server is displaying the contents of the directories. It should not be doing that. It is a security issue.

    You can find a fix here:

    Prevent Directory Listing - Web Site Security Tutorial | Tech-Evangelist

    Google will probably stop indexing the plugin directories if they start generating a status code 403 when an index file does not exist, which is what they should be doing.

    To use a robots.txt file, you'll need to have root access to your server.
    If you got this from a Google page, the writer doesn't know very much about web servers. Having root access to a server is something entirely different from placing a file in a site's root directory. Every site owner has FTP access to the root directory, but most hosting companies do not allow root access to the server.
    "Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote." -- Benjamin Franklin


  7. Thanked by:

    F-GD (17 September, 2009), sturat (17 September, 2009)

  8. #6
    Kovich's Avatar
    Kovich is offline Community Guardian
    Join Date
    Jan 2009
    Location
    Philadelphia, Pennsylvania
    Posts
    1,797
    Blog Entries
    30
    Thanks
    453
    Thanked 420 Times in 279 Posts
    Use the All In One SEO or Robots META Plugin for Wordpress and adjust the settings so that those pages are not indexed. Don't get rid of your robots.txt file, though - it's best to use both.

    Then login to the Google Webmaster's panel. Click Crawler Access and then Remove URLs. Enter all the indexed category URLs and Google will remove them.

  9. Thanked by:

    F-GD (17 September, 2009)

  10. #7
    dtkguy's Avatar
    dtkguy is offline Still Learnin'
    Join Date
    Dec 2008
    Posts
    391
    Thanks
    7
    Thanked 164 Times in 108 Posts
    Quote Originally Posted by TopDogger View Post
    ...
    If you got this from a Google page, the writer doesn't know very much about web servers. Having root access to a server is something entirely different from placing a file in a site's root directory. Every site owner has FTP access to the root directory, but most hosting companies do not allow root access to the server.
    I did (Removing my own content from Google) and yes I know and agree.

  11. Thanked by:

    F-GD (17 September, 2009)

  12. #8
    F-GD's Avatar
    F-GD is offline Lost Member
    Join Date
    Jan 2009
    Posts
    159
    Thanks
    65
    Thanked 24 Times in 17 Posts
    Thank you guys for the help. I dont see my files shown up....:s It got fixed automatically I guess

    My tags are set 'follow'. May be I should try to set them to 'no follow' but I am confused. I had a blog a while ago and the tags were indexing and I got most of the traffic from tags....

Similar Threads

  1. We need some suggestions for categories at CF. thanks
    By StephenM in forum Community Building
    Replies: 4
    Last Post: 17 February, 2010, 22:15 PM
  2. Google indexing time shocker
    By Dr. Stavros in forum Wordpress
    Replies: 7
    Last Post: 12 October, 2009, 16:26 PM
  3. Did Google Stop Indexing forums?
    By hendricius in forum On-Site SEO
    Replies: 7
    Last Post: 31 August, 2009, 13:37 PM
  4. Wordpress - Google Indexing discussions
    By Soulzripper in forum Wordpress
    Replies: 4
    Last Post: 10 June, 2009, 18:48 PM
  5. Replies: 8
    Last Post: 14 February, 2009, 12:28 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •