Make Your Wordpress 404 Pages NoIndex and NoFollow

**Will.Spencer** · 18 March, 2011, 19:05 PM

One of my sites had a software bug last year where two plugins could not get along nicely -- which resulted in about 100k nearly content-free pages being indexed by Google. These pages still show up in Google Webmaster Tools and that's starting to be annoying. They show up as being linked from search pages -- search pages which themselves no longer exist.

Google isn't being quick about forgetting about these pages, but here's a nifty trick that I am hoping will help. I have added the following code snipper to the <head> section of header.php:

Code:

<?php if( is_404() ) { ?>
 <meta name="robots" content="noindex,nofollow" />

<?php } ?>

This automatically adds noindex and nofollow to all of my 404 pages. I'll know in a few weeks whether it does the trick.

**TopDogger** · 20 March, 2011, 15:12 PM

Interesting approach.

Which two plugins created the problem?

I've seen pages that no longer exist linger in Google for up to four years, even when they are generating a valid status code 404. The problem is that Google has to check the page before it will see the robots tag or does anything about it. If GoogleBot no longer finds links with that URL, it can take some time. I hope yours get cleared out sooner than four years.

Keep up updated.

**Will.Spencer** · 24 March, 2011, 17:49 PM

Originally Posted by TopDogger

Which two plugins created the problem?

It was just a version mismatch -- upgrading one plugin caused an incompatibility until the other plugin released an upgrade.

Originally Posted by TopDogger

I've seen pages that no longer exist linger in Google for up to four years, even when they are generating a valid status code 404. The problem is that Google has to check the page before it will see the robots tag or does anything about it. If GoogleBot no longer finds links with that URL, it can take some time. I hope yours get cleared out sooner than four years.

Keep up updated.

After months of annoyance, my "Not Found" pages in Google Webmaster Tools fell from around 90k to just 13,446 -- and I am hoping for the rest to drop in the next few days.

My thinking is that this could be very important after the recent Farmer/Panda update.

**Shenron** · 24 March, 2011, 18:02 PM

This is indeed very interesting as I keep getting those for the same reason, but now I need to hire someone to update all my sites' header.php file...