Results 1 to 10 of 10

Thread: Blocked on robot txt

  1. #1
    thesyndicate is offline Newbie Net Builder
    Join Date
    Jul 2009
    Location
    NET<>DP
    Posts
    138
    Thanks
    14
    Thanked 5 Times in 4 Posts

    Blocked on robot txt

    I was reading this in a blog but i do not really understand it 100 %

    Always block pages (via robots.txt) from Google that contain search results as users don’t want to see another search page in Google search results and this may even invite a penalty.

    Should i block a page where a page search lands?

    like this

    We need to disallow /search-result.php/ ?


  2. #2
    5starpix's Avatar
    5starpix is offline Senior Net Builder
    Join Date
    Dec 2008
    Location
    Montreal, Quebec
    Posts
    1,710
    Blog Entries
    9
    Thanks
    158
    Thanked 227 Times in 164 Posts
    It would be a good idea to do that!

  3. #3
    thesyndicate is offline Newbie Net Builder
    Join Date
    Jul 2009
    Location
    NET<>DP
    Posts
    138
    Thanks
    14
    Thanked 5 Times in 4 Posts
    So i do not block the page with the search i block the page with the search result? How about sitemaps? Should i block them as well?

  4. #4
    5starpix's Avatar
    5starpix is offline Senior Net Builder
    Join Date
    Dec 2008
    Location
    Montreal, Quebec
    Posts
    1,710
    Blog Entries
    9
    Thanks
    158
    Thanked 227 Times in 164 Posts
    I don't see a reason why you should block your sitemap. You only want to block pages that you do not want to be indexed.

    You should have a sitemap.xml file though, but it is not a must.

  5. #5
    thesyndicate is offline Newbie Net Builder
    Join Date
    Jul 2009
    Location
    NET<>DP
    Posts
    138
    Thanks
    14
    Thanked 5 Times in 4 Posts
    Ok so i only block the search result pages and the admin pages then anything else?

  6. #6
    5starpix's Avatar
    5starpix is offline Senior Net Builder
    Join Date
    Dec 2008
    Location
    Montreal, Quebec
    Posts
    1,710
    Blog Entries
    9
    Thanks
    158
    Thanked 227 Times in 164 Posts
    If you have a non-unique privacy policy page, I would block that too

  7. #7
    thesyndicate is offline Newbie Net Builder
    Join Date
    Jul 2009
    Location
    NET<>DP
    Posts
    138
    Thanks
    14
    Thanked 5 Times in 4 Posts
    ahh good tip yes my privacy looks all the same on my pages, i just change the domain name. Is that enough? And do not google want to see you have a privacy?

  8. #8
    5starpix's Avatar
    5starpix is offline Senior Net Builder
    Join Date
    Dec 2008
    Location
    Montreal, Quebec
    Posts
    1,710
    Blog Entries
    9
    Thanks
    158
    Thanked 227 Times in 164 Posts
    Well if you have duplicate content in terms of privacy policy, then its best to block it

  9. #9
    Canonical is offline Unknown Net Builder
    Join Date
    Jul 2009
    Posts
    35
    Thanks
    0
    Thanked 19 Times in 11 Posts
    Always block pages (via robots.txt) from Google that contain search results as users don’t want to see another search page in Google search results and this may even invite a penalty.
    There is no way this is going to result in a penalty. That is total BS. probably 99.999999999% of the sites on the web that have an on-site search functionality do NOT block their search results pages from the crawlers. So whomever is telling you it can result in a penalty is jerking your chain.

    Actually, you can use search result pages on your site to your advantage from an SEO perspective by including "Most Popular Search" links on your site that link to search result pages for the most searched on topics. There are companies out there that sell this as an SEO product.

  10. #10
    thesyndicate is offline Newbie Net Builder
    Join Date
    Jul 2009
    Location
    NET<>DP
    Posts
    138
    Thanks
    14
    Thanked 5 Times in 4 Posts

Similar Threads

  1. Un-Blocked.me | Unblock Blocked Sites!
    By JettyZ in forum Proxy List Announcements
    Replies: 4
    Last Post: 29 January, 2011, 21:53 PM
  2. WP-Robot?
    By Alex01 in forum Wordpress
    Replies: 1
    Last Post: 16 July, 2009, 21:10 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •