Comments on: Another hungry java bot

by: frank

Mon, 02 Oct 2006 10:49:32 +0000

Why not blocking all javabots. It works for me like this:

SetEnvIfNoCase User-Agent “java/” bad_bot
Order Allow,Deny
Allow from all
Deny from env=bad_bot

(save as .htaccess)

I’ve noticed on my site that there are some spammers using the same ip-address over and over again. But the smart ones are using variable ip-addresses, these are the worst ones.
My question is how can you block these as they look like a normal browser?

by: Jo

Wed, 16 Aug 2006 09:13:58 +0000

is this for inside an .htaccess?

by: Martin Schuster

Tue, 18 Apr 2006 12:36:50 +0000

I guess you want to use
SetEnvIfNoCase User-Agent ^Java/1\. botsucker=yes
:)

by: Administrator

Sun, 19 Feb 2006 09:49:12 +0000

SetenvifNoCase User-Agent “Java/1.” botsucker=yes
SetenvifNoCase User-Agent “Snoopy” botsucker=yes
deny from env=botsucker

by: Ajay D'Souza

Sun, 19 Feb 2006 05:37:26 +0000

I’ve noticed a similar problem.

How do I block the useragent directly as opposed to just blocking the ips?

by: Joe

Wed, 15 Feb 2006 01:47:53 +0000

Alden is talking about stuff that has been around before the new personalized search options. Google has an Automatic URL Removal System. Submitting a URL won’t remove a page on its own, Google will quickly check (rather than waiting till your next site crawl) to see that the URL has been added to the site’s robots.txt and remove it. Since it checks for the site’s robots.txt, I doubt it could be used to maliciously remove a URL.

by: gpshewan

Tue, 14 Feb 2006 11:02:53 +0000

I took the decision a month or so ago to block all Java user agents. Can’t see the need for them, don’t see why any legit agent should be Java. I’d agree with Dirk on that … if somebody is using a feedreader that doesn’t say what it is in the UA and uses a stock ‘Java/x.x.x.’ field then tough.

Alden, the remove URL function is only a UI experiment (still?) and there’s no way for anybody to remove a sites URL from Google … just from their own search results. Google are the only ones who do the removing from the index

by: Alden Bates

Tue, 14 Feb 2006 05:33:10 +0000

If you do, avoid blocking 217.78.47.35, 216.239.3*.* and 66.102.6.136

When you use Google’s Remove URL function, they have a bot which polls the entered URL every so often using a Java user agent. If a site blocks all Java user agents, then it’s possible for a malicious party to get them removed from Google.

by: Dirk

Mon, 13 Feb 2006 21:06:09 +0000

Block it. The only legit requests from Java user agents we’re seeing are for our RSS feeds. In which case I’d say - though luck, use some other RSS reader.