Will.Spencer

It blocks every one I've seen. Each time an IP gets blocked by bot-trap, I get an email. I look at the email to see if the User-Agent is a bot that should not be blocked. If it is, I check it's IP address in WHOIS to make sure it's not some content thief faking their User-Agent. If the User-Agent and the WHOIS data match, I add the new IP address range to my white list.

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

TopDogger

I'm taking a look at this today.

If I understand this correctly, in addition to copying the other files to the server and modifying the robots.txt file, I need to set 777 permissions on the .htaccess file in the root directory and add the whitelist IPs. Is that correct?

I currently have the following list of spammer IPs blocked in the .htaccess on the test site. Do I just add the whitelist IPs to this block or keep them in a separate group?

Where is the blacklist being created? Could this system fill up the .htaccess file with blacklisted IPs over time?

Are you using a single pixel image for the link trap or did you place a larger image on the page somewhere?

BTW, the .htaccess protocol does not allow comments on the same line as a directive. I was making notations just like those in your whitelist for IPs that I was manually banning and my server error log filled up with error messages. You have to place comments on separate lines.

Will.Spencer

My .htaccess files seem to work with 644 (rw-r--r--) permissions, but that's because they are owned by the same user id as the web server runs under.

Hmmm... you got me... I think they can be separate.

It's appended to the end of .htaccess. The .htaccess can grow large over time. Every month or two I delete the older deny statements.

I'm using a single pixel image.

That's odd -- it seems to be working fine here under Apache 2.2. Are you running Apache 1.3 or Apache 2.2?

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

TopDogger

Thanks for the quick update.

644 is the standard permissions for the .htaccess. The instructions say to, "Make blacklist.dat and .htaccess writable by the web server user." I interpret that to mean that the file needs to be writable by the script being run by a user, which 644 doesn't cover. Maybe I'm interpreting this wrong. I'm looking at it as being similar to a cache directory, where the permissions typically need to be set to 666 or 777. Are the permissions on the blacklist.dat file set to 644 also?

I'm running Apache 2.2.10. One of the techs at my hosting company pointed that out a few months back while we were troubleshooting a server issue. He noticed that the Apache error log was pretty fat and packed with hundreds of messages pointing to the .htaccess files. He took a look at one of the .htaccess files and said that comments had to be on separate lines. I never saw an error with my sites, so the messages may have been warnings. It was something new to me.

Will.Spencer

I think it means "the script being run by the web server." On my system, the web server runs as the user www and .htaccess is owned by the user www.

But, 777 works too -- no matter who owns the file.

Very odd -- not a single mention of this in my error logs.

But I did some Googling and found other people with similar issues -- particularly when the comments contained forward slashes. So, it seems best to move the comments to separate lines.

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

TopDogger

OK. after I see how it logs the first spider, I will try setting the permissions to 644 to see what happens. If it does not work, an error will probably show up in the site's error log. Personally, I prefer not setting anything to 777.


All I use is a hash for comments in the .htaccess. There might be some kind of obscure server configuration issue that causes it to log errors on some servers, but not on others.

Will.Spencer

As an old Unix guy, 777 makes me feel weird.

The forward slashes were apparently being misinterpreted as being part of a CIDR (Classless Internet Domain Routing) statement. Like 192.168.0.0/24.

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

TopDogger

I tested it again this morning. It would not work with 644, so I had to settle for 666.

I do not understand how the CIDR number works. What range of IPs does 64.233.160.0/19 cover?

I have a long list of G spider IPs. Most are in different IP ranges.

Will.Spencer

CIDR notation uses binary math. /19 means "the first 19 binary digits are the network range and the rest is the IP address range." When you make the number after the slash larger, the networks get smaller. When you make the number after the slash smaller, the networks get larger.

10.0.0.0/8 is a traditional Class A network, i.e. 10.0.0.0 to 10.255.255.255.

192.168.0.0/24 is a traditional Class C network, i.e. 192.168.0.0 to 192.168.0.255.

Raising the number after the slash by one digit cuts the size of the network in two. Lowering the number after the slash doubles the size of the network.

I've never liked doing math, so I use a CIDR calculator for these calculations.

__________________
Submit Your Webmaster Related Web Sites to the NetBuilders Directory

TopDogger

I snagged my first spider yesterday.

I feel like I've gone fishing.

Will, are you running the spider trap link on multiple pages? Right now, I just have mine on the home page.