Blocked on robot txt

**thesyndicate** · 7 January, 2010, 01:21 AM

I was reading this in a blog but i do not really understand it 100 %

Always block pages (via robots.txt) from Google that contain search results as users don’t want to see another search page in Google search results and this may even invite a penalty.

Should i block a page where a page search lands?

like this

We need to disallow /search-result.php/ ?

**5starpix** · 7 January, 2010, 02:11 AM

It would be a good idea to do that!

**thesyndicate** · 7 January, 2010, 02:39 AM

So i do not block the page with the search i block the page with the search result? How about sitemaps? Should i block them as well?

**5starpix** · 7 January, 2010, 02:47 AM

I don't see a reason why you should block your sitemap. You only want to block pages that you do not want to be indexed.

You should have a sitemap.xml file though, but it is not a must.

**thesyndicate** · 7 January, 2010, 03:00 AM

Ok so i only block the search result pages and the admin pages then anything else?

**5starpix** · 7 January, 2010, 03:08 AM

If you have a non-unique privacy policy page, I would block that too

**thesyndicate** · 7 January, 2010, 03:38 AM

ahh good tip yes my privacy looks all the same on my pages, i just change the domain name. Is that enough? And do not google want to see you have a privacy?

**5starpix** · 7 January, 2010, 05:09 AM

Well if you have duplicate content in terms of privacy policy, then its best to block it

**Canonical** · 31 January, 2010, 18:06 PM

Always block pages (via robots.txt) from Google that contain search results as users don’t want to see another search page in Google search results and this may even invite a penalty.

There is no way this is going to result in a penalty. That is total BS. probably 99.999999999% of the sites on the web that have an on-site search functionality do NOT block their search results pages from the crawlers. So whomever is telling you it can result in a penalty is jerking your chain.

Actually, you can use search result pages on your site to your advantage from an SEO perspective by including "Most Popular Search" links on your site that link to search result pages for the most searched on topics. There are companies out there that sell this as an SEO product.