23 May, 2011, 18:58 PM
Need help, google bot going crazy
Since 3 days i have been hit by Google bot so strongly that it is consuming all my memory resources and giving 500 error. I dont know what is going wrong but google bot is accessing proxified pages, usually some twitter profiles continuously, i wrote the condition in robots.txt to exclude browse.php, but its not honoring that and keep giving hit on browse.php, i banned few ips of googlebot from .htaccess but it keep on coming back, it never happened before.
and for few reasons, yandex bot is doing same..hiting on proxified pages..
something wrong is going on..
Tags for this Thread