The normal way to do that would be to block them in robots.txt, but the bad guys just ignore robots.txt.
To deny them in .htaccess, you need to know their IP addresses, and the really bad guys change IP's frequently.
But it's not just bots. Web downloaders like wget, HTTrack, and WebCopier each up huge amounts of bandwidth and provide almost no value to the web site owner.
|