Bots that cheat
I get occasional visits from bots who advertise the site they’re coming from. That’s fine, as long as the URL is in the user agent. I appreciate the heads up, and I’m skeptical of those that don’t follow that practice. They’re a lot more likely to be banned.
But occasionally I find bots that not only have identifying information in the user agent, they also have a link to the site in the referrer field.
That’s what I call cheating.
And I intend to put some of those here:
- 70.242.210.41 - http://aweed.com - Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0 (root@aweed.com). Belongs to ID Domain Privacy Network (IDdp.Net)
- 212.56.39.2 - http://www.miragorobot.com/scripts/mrinfo.asp - HenryTheMiragoRobot (http://www.miragorobot.com/scripts/mrinfo.asp)
- 70.80.113.163 - http://www.avantbrowser.com - Avant Browser (http://www.avantbrowser.com)
- 18.85.45.85 - http://blogdex.net/about.asp - BlogBot/1.2
- 64.246.165.170 - http://www.whois.sc/ - SurveyBot/2.3 (Whois Source)
- 195.92.95.61 - http://www.netcraft.com/survey/ - Mozilla/4.0 (compatible; Netcraft Web Server Survey)
- xx.xx.xx.xx (normal user, munged) - Field blocked by Outpost Firewall (http://www.agnitum.com) - any user agent depending on end user’s browser
- 209.167.50.22 - www.seventwentyfour.com/ - LinkWalker
- 64.26.160.90 - www.brandimensions.com/ - BDFetch
- 208.237.254.167 - http://internetsupervision.com/urlmonitor - Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; internetsupervision.com)
- 69.150.87.183 - http://www.faxo.com - Faxobot/1.0
- 64.240.129.21 - http://iconsurf.com/ - IconSurf/2.0 favicon monitor (see http://iconsurf.com/robot.html)
- 84.233.148.20 - http://www.exabot.com/ - Exabot NG/MimeLive Client (convert/http/0.171)
- 64.207.46.75 - http://www.Syntryx.com/ - ANT Chassis 9.27; Mozilla/4.0 compatible crawler
- 128.240.229.65 - http://www.lemurconsulting.com/ - Mozilla/5.0 (compatible; LemSpider 0.1)
- 66.14.193.6 - http://www.blingblangblog.com - BlingBlangBlog.com Linkbot - (http://www.blingblangblog.com)
- 65.98.67.74 - http://www.fast-search-engine.com/ - User-Agent: Mozilla/4.0 (http://www.fast-search-engine.com/)
That’s an affair of the way you think of it. The referer is useful in the way to discover which websites have a link to yours. And if the bot comes and index your site, then the site of the people running the bot will, soon or yet, have a link to yours. Thus it is somewhat legitimate.
See, i don’t see the point of making public the statistics of your website. Statistics are useful for the admin! That’s ALL. I consider the fact of givin access to your site as childish as bragging about your car or income or number of female partners. “See the size of my referer-spam list? Do you see my big stats?”
User Agent string is meant to be the name of a bot, and it’s version. It should NOT provide information out of that. As a result, when you see a bot, out of googling the useragent, how could you know where the information it gathers will be published? That’s how the Referer is useful.
You’re wrong.
Many of these sites will never have a link to my site, and when they do, then THAT link is what’s legitimate in my logs, not the link to the site itself.
The user agent field of a bot should contain the URL of the site, but maybe shortened. The referer field of a bot should be blank.
[...] ct).
Sorry guys, that’s not how this works. Go searching for a clue …P.S. Ann Elisabeth wasn’t too happy with a previous incarnation of that bot either. [...]
I agree with the administrator of this site. Spam comments on blogs are one of the biggest problems, and any help you give on other your blog for user agents helps out all web masters. Thanks for the info.
What the heck is all this about? I found this ‘brandimentions’ thing in my stats page on my personal blog site on spaces.live.com. I don’t get it…call me stupid if you want but I’m just wondering what it all is?
Brandimensions call themselves “an online brand protection company” (direct quote from an email I received). So I assume their bot is spidering pages for trademark violation and similar things.
Here is the latest signature of the Spambot for http://www.fast-search-engine.com
Host: 65.75.152.120
*
/
Http Code: 200 Date: Nov 13 10:16:01 Http Version: HTTP/1.1 Size in Bytes: 4727
Referer: http://www.fast-search-engine.com/
Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
They are now using ^65.75.
I’ve alos blocked all of the IP’s that belong to the company that hosts http://www.fast-search-engine.com as they are well-known for Spam Bots
TheNerd, I dont really know if it’s anything worth bragging about. I mean, every single blog’s referrer report looks like this one. Not really uncommon.