Comments on: Suss out spam networks http://spamhuntress.com/2005/12/31/suss-out-spam-networks/ Just another WordPress weblog Mon, 13 Oct 2008 11:45:43 +0000 http://wordpress.org/?v=2.0.7 by: TrackBacks » Blog Archive » links for 2006-12-09 http://spamhuntress.com/2005/12/31/suss-out-spam-networks/#comment-74840 Sat, 09 Dec 2006 04:38:20 +0000 http://spamhuntress.com/2005/12/31/suss-out-spam-networks/#comment-74840 [...] Spamhuntress » Blog Archive » Suss out spam networks (tags: slogs a2b solution) [...] […] Spamhuntress » Blog Archive » Suss out spam networks (tags: slogs a2b solution) […]

]]> by: Hacks and Gadgets by HJL » Blog Archive » What (spam) sites are at that IP address? http://spamhuntress.com/2005/12/31/suss-out-spam-networks/#comment-2384 Sat, 14 Jan 2006 20:35:09 +0000 http://spamhuntress.com/2005/12/31/suss-out-spam-networks/#comment-2384 [...] sting services only shows some of the sites that are at their respective IP addresses. via SpamHuntress, see also comments there from Sam Critchely of A2B. [...] […] sting services only shows some of the sites that are at their respective IP addresses. via SpamHuntress, see also comments there from Sam Critchely of A2B. […]

]]> by: A2B and More » A2B blocklist makes it to Spam Huntress http://spamhuntress.com/2005/12/31/suss-out-spam-networks/#comment-2230 Mon, 02 Jan 2006 22:37:58 +0000 http://spamhuntress.com/2005/12/31/suss-out-spam-networks/#comment-2230 [...] 006 to Uncategorized Spam Huntress Ann Elisabeth has given our splog blocklist a mention on her blog. Thanks Ann Elisabeth! I left a comment explaining how we put the [...] […] 006 to Uncategorized
Spam Huntress Ann Elisabeth has given our splog blocklist a mention on her blog. Thanks Ann Elisabeth! I left a comment explaining how we put the […]

]]> by: Sam Critchley http://spamhuntress.com/2005/12/31/suss-out-spam-networks/#comment-2229 Mon, 02 Jan 2006 22:32:25 +0000 http://spamhuntress.com/2005/12/31/suss-out-spam-networks/#comment-2229 Hi Ann Elisabeth, Many thanks for making a mention of our splog blocklist on your blog. Thanks also for alerting us to the nofollow issue. ;-) I thought you and your readers might like to know how we pull the list of blocked IP addresses together. Firstly, at A2B (see http://www.a2b.cc for more) we run a search engine with a ping interface. Bloggers who have geo-located META tags in their HTML (see http://www.a2b.cc/help-searching-addurl-blogping.a2b for more) can ping us and we'll pick up (parse) their page and index it in our geosearch engine. We receive pings from many individual bloggers, a full ping feed from pingomatic.com, and bulk pings from several other sources, usually around 700,000 to 1 million pings per days. With approximately 200 IP addresses in the blocklist, about 37% of daily pings are blocked immediately. To generate the list, we recorded the URL of each website (read blog) we were pinged with and also converted it into the IP address of the web server for each URL. We recorded the IP address and added to a counter every time we received a ping for a URL on the same web server. We soon noticed that we were getting many thousands of pings for the same IP addresses, so pulled a script together to list the top IP addresses by number of pings. We built another script which showed all the URLs associated with each IP address. In order to decide which web server IP addresses to block we open this script and manually have a look at a random sampling of the URLs - it's usually pretty easy to tell if they're splogs as they're just full of advertising links or are quite random in their choice of subject matter. Any web server which has real blogs tends to stay off the blocklist (so that rules out blogspot.com even though people are using it for splogging). As soon as we'd blocked the first 112 IP addresses, the amount of traffic we were using parsing blogs dropped from 27GB per day (it was so high that it was costing us money in hosting charges) to 6GB/day. Of course, it began to creep up again soon after, so we're realising it's an ongoing effort and are beginning to think about blocking whole ranges of IP addresses. We're also going to put up a text-only list in IP address order - with any luck it'll be ready in the next day or so. I really hope that someone takes some action against sploggers on the hosted blogging services soon though, as it's costing us money on our hosting and it's a waste of time - we're a two-man show and don't have the time to keep writing scripts to fight it. Keep up the fight! Best wishes, Sam Hi Ann Elisabeth,

Many thanks for making a mention of our splog blocklist on your blog. Thanks also for alerting us to the nofollow issue. ;-)

I thought you and your readers might like to know how we pull the list of blocked IP addresses together.

Firstly, at A2B (see http://www.a2b.cc for more) we run a search engine with a ping interface. Bloggers who have geo-located META tags in their HTML (see http://www.a2b.cc/help-searching-addurl-blogping.a2b for more) can ping us and we’ll pick up (parse) their page and index it in our geosearch engine. We receive pings from many individual bloggers, a full ping feed from pingomatic.com, and bulk pings from several other sources, usually around 700,000 to 1 million pings per days. With approximately 200 IP addresses in the blocklist, about 37% of daily pings are blocked immediately.

To generate the list, we recorded the URL of each website (read blog) we were pinged with and also converted it into the IP address of the web server for each URL. We recorded the IP address and added to a counter every time we received a ping for a URL on the same web server. We soon noticed that we were getting many thousands of pings for the same IP addresses, so pulled a script together to list the top IP addresses by number of pings.

We built another script which showed all the URLs associated with each IP address. In order to decide which web server IP addresses to block we open this script and manually have a look at a random sampling of the URLs - it’s usually pretty easy to tell if they’re splogs as they’re just full of advertising links or are quite random in their choice of subject matter. Any web server which has real blogs tends to stay off the blocklist (so that rules out blogspot.com even though people are using it for splogging).

As soon as we’d blocked the first 112 IP addresses, the amount of traffic we were using parsing blogs dropped from 27GB per day (it was so high that it was costing us money in hosting charges) to 6GB/day. Of course, it began to creep up again soon after, so we’re realising it’s an ongoing effort and are beginning to think about blocking whole ranges of IP addresses.

We’re also going to put up a text-only list in IP address order - with any luck it’ll be ready in the next day or so.

I really hope that someone takes some action against sploggers on the hosted blogging services soon though, as it’s costing us money on our hosting and it’s a waste of time - we’re a two-man show and don’t have the time to keep writing scripts to fight it.

Keep up the fight! Best wishes, Sam

]]>