Comments on: Omni Explorer

by: Dan

Tue, 08 Aug 2006 18:06:03 +0000

this is my last attempt. Mod, please help me out.

if request.servervariables("REMOTE_ADDR") = "64.62.145.101" then
response.write(">html<" & vbcrlf)
response.write(">frameset<" & vbcrlf)
response.write(">frame src=’ht*p://www.vast.com/index.php?cb=" & Server.URLEncode(Now()) & "<" & "’" & vbcrlf)
response.write(">/frameset<" & vbcrlf)
response.write(">/html<" & vbcrlf)
response.end
end if

by: Dan

Tue, 08 Aug 2006 18:01:10 +0000

one more time?

if request.servervariables("REMOTE_ADDR") = "64.62.145.101" then
response.write("" & vbcrlf)
response.write("" & vbcrlf)
response.write("" & vbcrlf)
response.write("" & vbcrlf)
response.end
end if

by: Dan

Tue, 08 Aug 2006 17:57:39 +0000

I think I get it now - here goes:
” & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.end
end if
%>

by: Dan

Tue, 08 Aug 2006 17:56:40 +0000

My paste got chopped up in that last post - sorry.

Here’s the asp code I used:

” & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.end
end if
%>

by: Dan

Tue, 08 Aug 2006 17:55:12 +0000

I had a thought, and I’m wondering how this works…
I’ll let you know, but please leave me some feedback on it in the meantime.

I placed this at the top of the .asp pages that this spider was requesting most often:

” & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.write(”" & vbcrlf)
response.end
end if
%>

I’m hoping that this spider isn’t smart enough in itself to realize the duplicate links it’ll be creating by scanning a site derived from the crawler’s own data.

At the very least, I’m saving some bandwidth - the frameset only costs about 100 bytes, while my 403-forbidden page was three times the size.

Let me know what you think? And feel free to use the above, if you think it’s a good idea.

Dan

by: jood

Tue, 21 Mar 2006 16:52:45 +0000

Their robot crashed our site three times in as many weeks. Yesterday alone it logged almost a GIG in one single user session. I can see no reason why it would need to download multiple copies of our website at a time, and then repeat the process over a period of days.

We’ve blocked them now and written to them expressing our discontent, but I find it unlikely we’ll get a response. There are spam bot reports for this place all over the Internet.

by: Marian

Wed, 30 Nov 2005 12:50:49 +0000

The OmniExplorer bot has been on our site continuously since yesterday morning, but does not seem to be causing any bandwidth problem.
I don’t really understand why this bot is clearly getting people worried/hacked off. Could someone explain it in terms an IT thickie understands? Like what is likely to be the end result of its activities?

by: Derwood

Sun, 06 Nov 2005 05:01:34 +0000

I know this is an old article, but thought I’d just note that I just banned this bot from my server for pulling down data too fast and for ignoring robots.txt directives.. I sent them a nasty email, but after reading about this, I doubt it will do any good.

by: Cathy

Mon, 22 Aug 2005 21:48:13 +0000

I came across your site trying to find more info on some bastards causing my server to have seizures as they scrape my site. They use bogus user-agents and bogus referer strings. BUT ANYWAY.

I had issues with OmniExplorer months ago, but I loved that they had their info in the crap they left in my logs. I contacted them, and I’ve spoken with 3 people that work there. They were all very nice, and they all wanted to work with me to improve their bot’s manners. I do a lot of web sites for car dealers, who would like their information to display as many places as they can get it, so blocking OmniExplorer seemed a little harsh. I really liked those guys, and they helped me put stuff in my robots.txt file that would make their bot stop treading on my code where it shouldn’t (not part of standard robots.txt text, btw).

So anyway, what had been an ass pain at first really became not a problem at all after I talked to them.

by: ItaliaVista

Sat, 11 Jun 2005 04:28:44 +0000

Registered a Register.com those never reply to my email

Site and DN whois do not show any contact of course. This one have a really bad spike in bandwidth.

Hurricane Electric
760 Mission Court
Fremont, CA 94539

Tech Support 510 580-4100 support@he.net