Omni Explorer
Had a really bad spike in bandwidth, and chased down the cause. Veerry aggressive spider. Here’s what Webmasterworld has on it.
Short version: Possible bot used for building scraper directories. In other words, search engine spam.
Had a really bad spike in bandwidth, and chased down the cause. Veerry aggressive spider. Here’s what Webmasterworld has on it.
Short version: Possible bot used for building scraper directories. In other words, search engine spam.
May 21st, 2005 at 1:51 am
Can’t get to that article as I don’t really want to register with yet another site …
Anyway, beware that that bot sometimes comes around with a user agent Firefox/0.8. So to really block this bot, you should block the IP range.
I posted a little rant about that bot here: http://lists.geeklog.net/pipermail/geeklog-spam/2005-May/000007.html
May 23rd, 2005 at 12:12 pm
[…] 1.110 64.71.131.111 64.71.131.112 64.71.131.114 64.71.131.115 64.71.131.120 64.71.131.121 Earlier post about this bot This entry was posted o […]
June 2nd, 2005 at 1:47 am
Omni Explorer also has been hitting our site. Its very annoying as it removes variables from the query string and keeps making our sites error.
June 4th, 2005 at 4:14 pm
Had it hit my site at least twice today. Very bad spikes.
IP however different from the yours stated:
(so far)
65.19.169.242
65.19.169.248
June 9th, 2005 at 12:51 pm
OmniExplorer hit a site I admin today. Here’s the frequency distribution of IP addresses over 1.5 hours worth of attack:
2 65.19.150.212
2 65.19.150.234
6 65.19.150.233
13 65.19.150.230
15 65.19.150.211
57 65.19.150.213
78 65.19.150.207
979 65.19.150.241
1035 65.19.150.237
1998 65.19.150.246
3132 65.19.150.243
60080 65.19.150.242
I’ve blocked them by useragent on the reverse-caches, but I’ve got my eye on the class-C if they come back…
June 9th, 2005 at 3:35 pm
For ease of use, if you use the Cisco PIX firewall, you can add the following to block them:
object-group network omni-explorer.com
network-object 65.19.150.192 255.255.255.192
network-object 65.19.169.192 255.255.255.192
access-list acl-ext deny ip object-group omni-explorer.com any
Shane
June 10th, 2005 at 10:28 pm
Registered a Register.com those never reply to my email
Site and DN whois do not show any contact of course. This one have a really bad spike in bandwidth.
Hurricane Electric
760 Mission Court
Fremont, CA 94539
Tech Support 510 580-4100 support@he.net
August 22nd, 2005 at 3:48 pm
I came across your site trying to find more info on some bastards causing my server to have seizures as they scrape my site. They use bogus user-agents and bogus referer strings. BUT ANYWAY.
I had issues with OmniExplorer months ago, but I loved that they had their info in the crap they left in my logs. I contacted them, and I’ve spoken with 3 people that work there. They were all very nice, and they all wanted to work with me to improve their bot’s manners. I do a lot of web sites for car dealers, who would like their information to display as many places as they can get it, so blocking OmniExplorer seemed a little harsh. I really liked those guys, and they helped me put stuff in my robots.txt file that would make their bot stop treading on my code where it shouldn’t (not part of standard robots.txt text, btw).
So anyway, what had been an ass pain at first really became not a problem at all after I talked to them.
November 5th, 2005 at 11:01 pm
I know this is an old article, but thought I’d just note that I just banned this bot from my server for pulling down data too fast and for ignoring robots.txt directives.. I sent them a nasty email, but after reading about this, I doubt it will do any good.
November 30th, 2005 at 6:50 am
The OmniExplorer bot has been on our site continuously since yesterday morning, but does not seem to be causing any bandwidth problem.
I don’t really understand why this bot is clearly getting people worried/hacked off. Could someone explain it in terms an IT thickie understands? Like what is likely to be the end result of its activities?
March 21st, 2006 at 10:52 am
Their robot crashed our site three times in as many weeks. Yesterday alone it logged almost a GIG in one single user session. I can see no reason why it would need to download multiple copies of our website at a time, and then repeat the process over a period of days.
We’ve blocked them now and written to them expressing our discontent, but I find it unlikely we’ll get a response. There are spam bot reports for this place all over the Internet.
August 8th, 2006 at 11:55 am
I had a thought, and I’m wondering how this works…
I’ll let you know, but please leave me some feedback on it in the meantime.
I placed this at the top of the .asp pages that this spider was requesting most often:
” & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.end
end if
%>
I’m hoping that this spider isn’t smart enough in itself to realize the duplicate links it’ll be creating by scanning a site derived from the crawler’s own data.
At the very least, I’m saving some bandwidth - the frameset only costs about 100 bytes, while my 403-forbidden page was three times the size.
Let me know what you think? And feel free to use the above, if you think it’s a good idea.
Dan
August 8th, 2006 at 11:56 am
My paste got chopped up in that last post - sorry.
Here’s the asp code I used:
” & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.end
end if
%>
August 8th, 2006 at 11:57 am
I think I get it now - here goes:
” & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.end
end if
%>
August 8th, 2006 at 12:01 pm
one more time?
if request.servervariables("REMOTE_ADDR") = "64.62.145.101" then
response.write("" & vbcrlf)
response.write("" & vbcrlf)
response.write("" & vbcrlf)
response.write("" & vbcrlf)
response.end
end if
August 8th, 2006 at 12:06 pm
this is my last attempt. Mod, please help me out.
if request.servervariables("REMOTE_ADDR") = "64.62.145.101" then
response.write(">html<" & vbcrlf)
response.write(">frameset<" & vbcrlf)
response.write(">frame src=’ht*p://www.vast.com/index.php?cb=" & Server.URLEncode(Now()) & "<" & "’" & vbcrlf)
response.write(">/frameset<" & vbcrlf)
response.write(">/html<" & vbcrlf)
response.end
end if