Growing trend for blank user agent
I noticed some trackbacks with a blank user agent again, and decided to check how many accesses I had with blank user agent and referrer. And I’m sad to say, the number is growing.
Here are some examples:
- A Firstmile IP number (Norway) fetches my blog three times a day. A bot.
- Someone from Holland has a feed reader without a user agent. Regular reader.
- Someone using the Inflow ISP is fetching the blog once a day. A bot.
- Wordpress installations send pings with a user agents. But there are a few accesses without user agents. Some HEAD, some GET.
- A Swede (bredbandsbolaget) came by, suppressing user agent and referrer
I’d love to block users without referrer and user agent, but that might be premature.
Did I remember to tell you I’m sick of software that doesn’t identify itself?
May 1st, 2005 at 11:20 am
I block requests with blank User-Agents *and* referrers from sensitive parts of my web site (mailing list archives, blog). Blocking it for all pages seems would feel a bit paranoid.
May 1st, 2005 at 4:06 pm
It would be great if we could block blank User-Agents, that would force software to be more informative, but it would only help with spam for a short while. Even though most spammers are dumb they do adapt given enough time. The few that don’t already would just start faking the User-Agent. Then you would only be blocking legitimate users with bad software. Some probably send blank User-Agents on purpose. I know that is a privacy option with JunkBuster.
The good part is that some languages don’t make changing the User-Agent easy. I wrote a simple search engine scraper in Java and had a tough time figuring out how to fake it. Most search engines would block or send funny results when they saw a Java User-Agent. I didn’t know Java well, but I doubt most spammers are highly skilled programmers either.