Thread: how to block all spider ?

Guy how to block all spider with htaccess ???
yes everyone include all major search engine, cause i setup a few live test site with huge content but prefer not showing it to any bot

i try wp stop all SE setting, but google still index the site, with title and no description ...

thanks

I would suggest adding entries in your htaccess files


i wrote a article for simmilar task not exctly all bots.

however with bit of tweaking this should work.

htaccess based spamBot and Leacher Blocking Code | Anant Shrivastava : Techno Enthusiast

Bad spiders will ignore the robots.txt file. Only legitimate spiders use the robots.txt.

For bad spiders, the best solution is to forcefully block them using the .htaccess file.

You can use the robots.txt file to block Google, Yahoo, Bing/MSN and others. Just use the following in the file.

Code:

User-agent: * 
Disallow: /

Popup blocker?

Hahahahhahaha....

Thanks, I needed a laugh!

@SonnyCooL

You need to use robots.txt like TopDogger wrote. Then you should place meta tag:
<meta name="robots" content="noindex, nofollow">

This "nofollow" is not necessary in your case. You can change it with "follow". For bad spiders, use your .htaccess to block them

Hey Sonny .. If your IP address is static, then you could set up a rule in a .htaccess file to allow only your IP. Only problem is that if your IP is dynamic then you would have to update the rule each time it changes.

This will automatically block everyone and you could also redirect them to a coming soon / maintennance page etc until the site is ready.

Exclude your IP only and give eveyone else a 403 forbidden.

Code:

RewriteCond %{REMOTE_ADDR} !^111\.222\.33\.4
RewriteRule ^(.*)$ - [F,L]

Exclude your IP and 302 redirect everyone else to a temp holding page such as a index.html.

Code:

RewriteCond %{REMOTE_ADDR} !^111\.222\.33\.4
RewriteCond %{REQUEST_URI} !/index\.html$ [NC]
RewriteRule ^(.*)$ /index\.html [R=302,L]

There's way too many bots to block them all in a htaccess. You would have better results using some kind of anti-bot script.

After a long time, encountered a very good question. There are two methods to block spiders through .htaccess.

1. Use SetEnv directive with combination with FilesMatch.

SetEnvIfNoCase user-agent "^Custo" bad_bot=1
SetEnvIfNoCase user-agent "^Bot\ mailto:craftbot@yahoo.com" bad_bot=1
<FilesMatch "(.*)">
Order Allow,Deny
Allow from all
Deny from env=bad_bot
</FilesMatch>

Just define all the bad bots with SetEnvIfNoCase. The


2. ModRewrite Method

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
RewriteRule ^.* - [F,L]