Robots.txt?

**dmi** · 24 July, 2009, 10:45 AM

This is how my robots.txt looks like now:

Code:

Sitemap: http://www.mysite.com/sitemap.xml.gz

User-agent: *
Disallow: /contact/
Disallow: /faq/
Disallow: /privacy-policy/
Disallow: /terms/
Disallow: /feed/
Disallow: /wp-
Disallow: /cgi-bin/
Allow: /wp-content/uploads/

User-agent: Mediapartners-Google
Allow: /

User-agent: duggmirror
Disallow: /

Any suggestions?

Now, on my site, the links to those blocked pages are "nofollow". Should I remove the nofollow, because I blocked them in robots.txt? To clarify: the links to these pages are in my footer, and they look like this:

<a href="/page/" rel="nofollow">Page</a>

So, would you remove the nofollow from there? I want to preserve the PR.

**Will.Spencer** · 25 July, 2009, 05:15 AM

Originally Posted by dmi

Now, on my site, the links to those blocked pages are "nofollow". Should I remove the nofollow, because I blocked them in robots.txt? To clarify: the links to these pages are in my footer, and they look like this:

<a href="/page/" rel="nofollow">Page</a>

So, would you remove the nofollow from there? I want to preserve the PR.

The only way to preserve the PageRank is to allow Google to follow those pages, which means allowing them in robots.txt.

If you block them with robots.txt or with nofollow, Google won't be able to count the links from those pages to your navigational elements.

The tag you're looking for is noindex, but I don't know how to put that into the meta tags of individual Wordpress pages.

Of course, we're talking about a very small PR leak here.

**Diesel** · 1 September, 2009, 18:56 PM

robots.txt is a very important file if you want your website to rank well on search engines. Before editing robots.txt know it well to make sure that it doesn't contain errors.

**5starpix** · 1 September, 2009, 22:49 PM

Do you guys know any specific sites to obtain a good robots.txt file?

**Alastor** · 1 September, 2009, 23:27 PM

Originally Posted by Diesel

robots.txt is a very important file if you want your website to rank well on search engines. Before editing robots.txt know it well to make sure that it doesn't contain errors.

Your not just talking, I screwed a site up bad that way and it took over 2 months to recover from my mistake.