Thread: Need help, google bot going crazy

Hello,
Since 3 days i have been hit by Google bot so strongly that it is consuming all my memory resources and giving 500 error. I dont know what is going wrong but google bot is accessing proxified pages, usually some twitter profiles continuously, i wrote the condition in robots.txt to exclude browse.php, but its not honoring that and keep giving hit on browse.php, i banned few ips of googlebot from .htaccess but it keep on coming back, it never happened before.

and for few reasons, yandex bot is doing same..hiting on proxified pages..

something wrong is going on..

Please help

Can you show us exactly what your robots.txt file looks like?

A temporary fix might be to rename your browse.php file if you can. You will need to change the name referenced in /includes/init.php on line 25 "define('SCRIPT_NAME', 'browse.php');" that might at least buy you some time if it's automated. It's something I do anyway to try to delay detection by filtering networks, well worth doing imho. Don't forget to change any quick links or pre-made proxify shortcuts to the new name too. You could even make a new browse.php file that redirects to bing, lol. It may be a faked referral id, so blocking google might not stop the attacks anyway.

Originally Posted by NickW

Can you show us exactly what your robots.txt file looks like?

User-agent: * Disallow: /browse.php User-agent: Googlebot-Mobile Disallow: /

Usually Googlebots follow the robots.txt. If you didn't check already, maybe have a quick look if the IP addresses actually belong to google and not someone else.

Here's something you can use to block that s#!t until you get it sorted out.

PHP Code:

$ua = $_SERVER['HTTP_USER_AGENT'];
$detect = array('googlebot','yandex');

if(!empty($ua))
{
  foreach($detect as $d)
  {
     if(stripos($ua,$d) !== false)
     {
        exit("<strong>$d</strong> detected - No proxy for you !");
     }
  }
} 


If you add this to the top of your browse.php page it should stop the browse page from loading and restrict access to 'anyone' trying to go through your proxy with a googlebot or yandex useragent.

You could also send them to the default Glype banned page:

PHP Code:

header('HTTP/1.1 403 Forbidden', true, 403);
      echo loadTemplate('banned.page');
      exit; 


or redirect them to another page with a message if you'd prefer to do something else to/with them.

PHP Code:

exit(header('Location: ./some-other-page.php')); 


If you use it, let me know if you have any problems.
I added a little more info in this guide on How to block bots from accessing pages through your proxy .

Your issue is surely not with Google, but a bad bot pretending to be Google. There are a lot of bad bots(BlockScript tracks over 643,228 bad bot IPs) and they won't respect your robots.txt file.

Originally Posted by Kaustubh

User-agent: * Disallow: /browse.php User-agent: Googlebot-Mobile Disallow: /

Umm... that's "Googlebot-Mobile", not "Googlebot".

Alright, the problem was coz of googlebot was stucked in some kind of loop.