Spamhuntress

Ajay asked me to teach him to figure out spammers based on the logs, like I do.

Well, Ajay, I imagine others want to do that as well, so here’s a post about it.

But I can’t promise you’ll figure it out. It’s got to do with how your brain processes information as well as knowing what to look for.

First of all, you need access to the raw logs. I’m not talking about Latest Visitors, like you’ll find in cpanel. I’m talking about the full raw logs. Preferably going back to the start of the month. And you should save them at the end of each month. Save them to your computer in a zipped or gzipped format (that compresses them to about one tenth of the size).

If you’ve got hosting that allows you ssh (command line interface) access, you can use grep (linux command set) on the raw logs while they’re still on the webhost. If you don’t (most budget webhosting doesn’t allow ssh), then you need to download the logs to your computer.

The best tool I’ve found for dealing with raw logs on a Windows computer, is TextHarvest. It’s a free demo app. It’s plenty powerful enough for our purposes, but also shows how powerful the proprietary paid versions will be for other kinds of tasks.

When I use TextHarvest, I start by unzipping the log file and navigating to it, so it’s my input file. I then make a bookmark for the output file in my browser, and untick the autoview (the app will crash if you autoview several megabytes of log file).

Then, when I receive a comment I suspect to be from a jokester, like that comment from Dima, I find the IP address from the comment, and input that in the Keep list. And I’ve found that using the \ slash instead of the / (which is the default they tell you to use, but any character will work) one makes sense, since I often search for user agents as well.

Press start, and it chugs through the log file.

When I then look at the log output, it contains only the entries made by that IP number, and possibly when others look at their user page if they’ve made any wiki edits.

What I then look at, is if the pattern is that of a normal user. Does it load any images? Does the browsing history look legit? There are so many things jokesters and spammers do, so I won’t detail them here. But YOU need to figure out what a normal browsing history looks like on YOUR website. And if some user diverts from that, what’s the reason? Sometimes normal users don’t have normal browsing patterns either.

When you look at the browsing history, pay attention to the user agents. Spammers sometimes have scripts that change their user agents more or less in mid flight.

Finally, a warning: Don’t be too cocky now that I’ve taught you the basics of checking logs. There’s a LOT more to be learned, and I’ve seen people completely misunderstand the information in front of them. Hopefully I haven’t made too many errors myself. I started reading logs years before I became Spamhuntress, so I had a head start.

This entry was posted on Sunday, August 20th, 2006 at 2:58 am and is filed under Tools. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.