Can you show us exactly what your robots.txt file looks like?
Since 3 days i have been hit by Google bot so strongly that it is consuming all my memory resources and giving 500 error. I dont know what is going wrong but google bot is accessing proxified pages, usually some twitter profiles continuously, i wrote the condition in robots.txt to exclude browse.php, but its not honoring that and keep giving hit on browse.php, i banned few ips of googlebot from .htaccess but it keep on coming back, it never happened before.
and for few reasons, yandex bot is doing same..hiting on proxified pages..
something wrong is going on..
A temporary fix might be to rename your browse.php file if you can. You will need to change the name referenced in /includes/init.php on line 25 "define('SCRIPT_NAME', 'browse.php');" that might at least buy you some time if it's automated. It's something I do anyway to try to delay detection by filtering networks, well worth doing imho. Don't forget to change any quick links or pre-made proxify shortcuts to the new name too. You could even make a new browse.php file that redirects to bing, lol. It may be a faked referral id, so blocking google might not stop the attacks anyway.
If I can't be a good example, I'll just have to be a terrible warning...
Usually Googlebots follow the robots.txt. If you didn't check already, maybe have a quick look if the IP addresses actually belong to google and not someone else.
Here's something you can use to block that s#!t until you get it sorted out.
If you add this to the top of your browse.php page it should stop the browse page from loading and restrict access to 'anyone' trying to go through your proxy with a googlebot or yandex useragent.PHP Code:
$ua = $_SERVER['HTTP_USER_AGENT'];
$detect = array('googlebot','yandex');
foreach($detect as $d)
if(stripos($ua,$d) !== false)
exit("<strong>$d</strong> detected - No proxy for you !");
You could also send them to the default Glype banned page:
or redirect them to another page with a message if you'd prefer to do something else to/with them.PHP Code:
header('HTTP/1.1 403 Forbidden', true, 403);
If you use it, let me know if you have any problems.PHP Code:
I added a little more info in this guide on How to block bots from accessing pages through your proxy .
Last edited by Mike-XS; 24 May, 2011 at 14:41 PM.
Submit Your Webmaster Related Sites to the NB Directory
I swear, by my life and my love of it, that I will never live for the sake of another man, nor ask another man to live for mine.
Alright, the problem was coz of googlebot was stucked in some kind of loop.