Omni-explorer

From Spamhuntress

Jump to: navigation, search

Update: (Aug 22 2005) I caught this bot yesterday deep crawling without referrers like it used to have previously (version 3.28)

Update: There's a new statement on their website. An apology for how hungry it was, along with an assurance it'll obey robots.txt

Problem is, it's still hungry, still way too fast. It came by June 28, started with robots.txt, then gobbled up almost 6,4 MG, and was done with hitting 785 pages (no images or helper pages) in 14 minutes.

Now tell me why I shouldn't block it (in robots.txt)?

Omni-explorer, you need to do better than that measly explanation!

Latest IP number was 65.19.150.249

Update Dirk caught it ignoring a robots.txt block


The Omni Explorer bot has caused a fair bit of outcry lately in 2005.

It can deep crawl a big website in a short time, downloading several hundred megabytes in one sitting.

This bot is unique in that it always carries referrers, whether they're from your site or someone else's. These referrers can be mistaken for referrer spam. So if you find referrers, you need to check if they're legit. All the referrers in my logs were legit. Update: Effective June 21, it no longer carries referrers. On my suggestion, I might add...

Doesn't even touch robots.txt. Update: First hit on my robots.txt June 20.

It switches IP ranges often, and seems to unleash a number of machines on one site at the same time. So far all the IP ranges have been from Hurricane Electric

So, what and who is Omni Explorer?

The website used to say "comming soon_" centered in black on white background. They later corrected it to "coming soon_". As of June 15, 2005 the site states "Omni-Explorer is a stealth-mode venture-backed startup based in Silicon Valley. Stay tuned to this site; we plan on launching shortly." There is also an apology for their bot behaving badly in the past.

They've offered a personal search bot for few years back, on botscape.com (216.127.66.63 on EV1): 

Borislav Agapiev
10857 NW Appellate Way
Portland, OR 97229
US
Phone: 503-646-8973
Email: boris@omni-explorer.com

The company is named Omni-Explorer Technologies, and is situated at 

6700 SW  105th
Beaverton, OR 97229
Phone: 503-478-9696

That's the address in the whois of omni-explore.com and most of the other sites. Many companies have offices in that building. Either that, or it's a mail box location. I can't tell.

The IP number 216.40.249.17 on EV1 hosts several other websites, and they're owned by the same company, with this figurehead:

Borislav Agapiev

Most of the sites are fed by the spider bot, such as job listings and car sales listings. So if you've got a site in one of those fields, they may be scraping your site, to offer your content to their customers. You decide if that's positive or negative for your company.

They also have an online game, for virtual selling of cars. The same bot that's feeding the car sales site, is probably feeding that site.

There's a German site, offering much the same content. For instance autoschnueffler.de and product-explorer.com. Someone from 84.154.163.108 (Germany) tried removing this paragraph... The whois for omni-explorer.de is: 

Name: 	Torsten Heissler
Adresse: 	weblift.de Steinerweg 27
PLZ: 	78239
Stadt: 	Rielasingen-Worblingen

Omni-Explorer were searching for workers in Serbia April 2005. Java programmer, Analyst and PHP programmer. Either oursourced to a location in Serbia, or that's their real location. One of their employees appear to be named Nenad Trickovic. He posted in search of Serbian programmers.

User agents (I've munged them - adding an asterisk - to avoid hyperlinking): 

OmniExplorer_Bot/3.28 (+h*tp://www.omni-explorer.com) WorldIndexer
OmniExplorer_Bot/1.07 (+h*tp://www.omni-explorer.com) Internet Categorizer
OmniExplorer_Bot/1.10 (+h*tp://www.omni-explorer.com) Jobs Crawler
OmniExplorer_Bot/1.09 (+h*tp://www.omni-explorer.com) Cars Crawler
OmniExplorer_Bot/1.09 (+h*tp://www.omni-explorer.com) personals Crawler


Old posts


Other posts

Retrieved from "http://spamhuntress.com/w/index.php/Omni-explorer"
Personal tools