Comments on: Omni Explorer gobbles 300 megabytes

by: gotit all undercontol

Sun, 26 Nov 2006 22:11:32 +0000

We have been blacklisting and null routing all traffic that originates from hurricane electric for over two years and have not had one single complaint from a valid sender. Therefore we concluded there are no non-spammers on he.net networks…

by: Irate Webmaster

Sun, 29 Jan 2006 01:52:46 +0000

I just had a visit on 1/22/06 and 1/23/06 from 65.19.150.248

It gobbled 3GB in bandwidth over 61,000+ page hits in those 2 days.

My robots.txt file wasn’t followed.

Just thought I’d share.

by: James Robinson

Sat, 03 Dec 2005 19:08:55 +0000

Just got through blocking them via firewall rules since they wandered into robots.txt restricted space. It was nice of ‘em to post their IP ranges on their site, but now after googling about them and learning just how evil they really are, I wouldn’t be surprised if they happened to omit a few blocks on that list.

I wrote their feedback email address telling ‘em that their robot didn’t honor robots.txt properly [ didn’t honor “User-agent: *” block at all ], and I got a pair of emails from an ‘Anton Stanley’ saying that he apologized, that their robot was having troubles lately, and that they were planning on launching several ‘vertical search engines’ by the end of the year, and they could drive quality traffic to our site. And he asked if we’d just go back to blocking ‘em via a user-agent-specific robots.txt block. The answer was no. I hope their robot stumbles gets gummed up bad when their packets are routed to /dev/null .

by: Hankwang

Tue, 21 Jun 2005 21:47:46 +0000

(Moderator, please delete previous comment, I messed up the html tags).

To Chris about the honeypots: that’s basically what I did except for the automated IP blocking. The hidden link to the honeypot is the first link on each of my pages. I do it like this:

to discourage the search engines from ranking my pages based on what’s in the link text (the word “email”). The nofollow will prevent Google from indexing the linked urls (even if it is not crawled due to robots.txt, Google will index the URL if there are links pointing to it).

by: Administrator

Mon, 20 Jun 2005 23:24:13 +0000

They say it’s going to obey robots.txt from now on. So read up on that.

by: Anonymous

Mon, 20 Jun 2005 23:18:25 +0000

Hi
I am newbie to blocking a bot. This bot has been gulping more bandwidth on my site. Can anybody please give me instruction on how to block this bot with blocking code for robot.txt
I have two omni bot on my website
OmniExplorer_Bot/1.07 (+http://www.omni-explorer.com) Interne
OmniExplorer_Bot/1.10 (+http://www.omni-explorer.com) Jobs Cr
Thanks in advance.

by: Administrator

Sun, 19 Jun 2005 13:17:43 +0000

They’ve just put up a notice on their site. Apologizing for how hungry the bot was, and saying it will obey robots.txt.

There’s no word about why it sometimes comes with a user agent that looks like a regular browser.

Frankly, I believe they need to include more than what they have. They need to include the IP range, so we can see who’s them and who’s a copycat.

by: chris holland

Sat, 18 Jun 2005 22:56:13 +0000

ugh sorry i guess the html i had in my post did get literally rendered, what i meant to say about javascript is add the follwing attribute to your “a” link tags: onmouseover=”this.href=’about:blank’;”

by: chris holland

Sat, 18 Jun 2005 22:54:00 +0000

To kinda build on-top of what some other posters mentioned, what would guys think of this automated process:

1) set-up honeypot web pages on your site. Those are pages no valid end-user or valid bot should ever hit.

2) place prominent links to those honeypot pages everywhere on your *legit* pages. But hide those links using CSS after giving them a class attribute. you use display:none; and/or visibility:hidden. You could even use javascript to nullify the urls, so a honeypot link might look like: click me with a css block somewhere that declares: .honeypotStyle {display:none;visibility:hidden;}. I’m not sure whether that’ll handle mobile devices too well. You might also check for http headers being sent. A normal browser will send a slew of http headers such as the “Accept:” one. If a bot author is lazy/ignorant, he won’t send those.

3) place the honeypot URLs in a robots.txt file at the root of your site, be sure to deny all. From here, all legit bots will ignore your honeypots urls.

4) set-up an automated process by which an IP address gets blacklisted/thoroughly logged with ALL http headers that were sent in the request/reported/let-your-imagination-run-wild as soon as it hits your honeypot url.

Would this work?

-chris

by: Thomas Anderson

Wed, 15 Jun 2005 09:09:58 +0000

This list may be easier to read than the last:

Denied IP Listing and Netmask for Hurricane Electric
216.218.130.136 255.255.255.248
216.218.128.0 255.255.128
64.71.128.0 255.255.192.0
65.19.128.0 255.255.192.0
66.220.0.0 255.255.224.0
209.51.160.0 255.255.224.0
216.66.0.0 255.255.224.0
64.62.128.0 255.255.128
66.160.128.0 255.255.128
216.218.130.128 255.255.255.248
64.71.191.56 255.255.255.248
66.220.4.240 255.255.255.240
216.218.229.80 255.255.255.240
216.218.158.16 255.255.255.240