How to find a compromised script on shared virtual webhost
That debacle with *** webhost got me thinking. What’s the best way of figuring out the compromised script on a shared virtual webhost?
Say the spamming is referrer spamming.
The spamming could be done in any of the following ways:
1) User uploads spam script to his user area
2) A compromised script is used with a functionality like a proxy server
3) The server is compromised, and the spammer has installed a script that’s spamming
What’s the one thing you can count on in that situation? The offending traffic is outbound. It’s coming from any port, but connecting to port 80 on the remote system (the website they’re connecting to, so it appears in access logs).
But the spamming could be controlled by inbound traffic, or by a script on the server.
What would be the best tools to narrow down to which script is doing the spamming, or matching inbound with outbound traffic?
July 29th, 2006 at 5:22 am
A spam script can easily inserted by scanning hosts for vulnerable scripts /content management systems. It may take a couple of weeks if not months before these scripts will be used (waiting till the dust has settled). Also, if a spammer isn’t silly, he won’t select servers which only host a few domains, but rather go with overcrowded servers run by cheapo hosting companies. This is a strategy used by many phishing sites, rather than registering domains, which would require a bullet proof hosting company, which likely is already on someone’s black list.
Unlike inbound traffic (which will be recorded in server logs) outbound traffic remains uncontrolled, so it’s quite a lot of work to determine whose script out of hundreds of virtual hosts is actually responsible for spamming. But you may as well never see the source of the spam as it’s pretty trivial for an experienced programmer to have his spam tool used proxy server. And if those proxies are located in mainland China or Korea, your odds at finding someone telling you who’s actually responsible for the spam are fairly slim.
Actually you could even realise a script, that automatically updates its proxy list by contacting a remote server (ideally bulletproof like netcathosting or inhoster). Since that’s again outbound, you won’t find any records of the connection. Silly scripts would require manual interaction, which means the spammer needed to contact the server and insert the update -> inbound connection -> present in server logs. But then you’ll have the problem of finding the compromised virtual host out of hundreds on the server.
So as a summary, your luck in squashing a spammer largely depends on his stupidity
July 29th, 2006 at 5:36 am
A small correction of my comment:
“…Actually you could even realise a script, that automatically updates its proxy list by contacting a remote server…”
This wasn’t very thought well by me
. A smarter way of updating proxies would be to contact a couple of proxy sites, fetch their updates and extract the servers replacing the old list.
July 29th, 2006 at 5:51 am
To Vasily:
How about a webhost, after being notified that something like this is happening, wants to monitor outbound connections. These will look different from responses to inbound connections. So it should be possible to monitor them, at least for a while. Have you thought out exactly what would be needed?
Ethereal with some filter rules?
How much load increase would running Ethereal in this way introduce?
BTW, clever nickname. A rewrite on the old Russian John Doe!
July 31st, 2006 at 7:54 am
An easy, but fairly boring pseudo fix is, if one is runing a Nuke derivitive (phpNuke, PostNuke, Xooops, etc.), disallow all html and tags in posts. I know this is defeatable, but it is a fine way to make ye ole’ spammer work harder.
July 31st, 2006 at 2:24 pm
Hey Not-Not, fancy seeing you here
Are you talking about comment spamming here? Because disallowing html and tags is a fine thing, but would it help with securing scripts against injection and other misuses?
August 1st, 2006 at 9:53 am
From what I hastily read it seemed like you were wondering how to remedy a compromised script and/or how to track down said script. It seems that Blog/CMS type systems many times fall prey to compromised scripts because they allow posts with tags (think AJAX here) OR because they allow posts with header redirect tags that fire spam runs on a target machine elsewhere. If, on the other hand you were talking about a script that was compromised via an OS path (think rootkit) then I’d say the box is Pwn3d anyway and the problem resides with a lackadaisical sys admin. Said sys admin should, of course, be barred from cheap beer distribution for at least 48 hours as penance for her/his transgression
. If the CMS/Blog install was compromised via a file upload allowed by the software interface / web interface the person running CMS/Blog has chosen, well the person running CMS/Blog should be barred from cheap beer distribution AND spatula use for at least 48 hours as penance for her/his transgression 
. Insofar as connecting to port 80 on the remote machine, maybe and maybe not. One can run a web server or java application container on any port not already bound to by another process. One way to keep this kind of thing under wraps is to smear on a healthy dose of IpTables to allow only in and out bound traffic on ports to which the sys admin agrees. For example, if there were a box called, err hemm, the “mighty amigo” the SA might set up the IpTables to only allow inbound traffic on TCP-22, TCP-80, TCP-25 and nothing else. Further, said SA might also only allow outbound connections on the same ports. If a user on the “mighty Amigo” wanted a daemon to run on some other port, they’d have to clear that with the SA. This way even if the compromised script tried to form its connection on 80 or even some other port you’d see it in the syslog. Insofar as inbound “zombie” control traffic … that’s a bit more difficult. The reason being that the spam controller already knows you’re compromised and, if they’re smart, they’ve already figured out what your TCP allowed infrastructure looks like. So, in this case one could use mod_throttle to choke inbound traffic the apache instance. This is because a web server should be doing most of its work going outbound anyway (small inbound requests, fat outbound replies). If these were not the paths of compromise you intended then pay no attention to me.
Not-Not
August 1st, 2006 at 11:16 am
Hi again Not-Not,
I have (to start with) no idea what kind of “ship” the sys admin in question is running. Let’s just say I don’t think he’s on first name terms with and IDS of any color. They’re probably running cpanel or some other control panel, and have up to 100 users on one box. Webhosting.info shows 480 domains on that specific box (though that data is out of sync - it might be significantly less or more). Loads of them look spammy too, so management is probably lax to start with.
But the problem that host is facing could be faced by any webhost that sells virtual webhosting, if they don’t know what they’re doing.
The box you’re describing sounds like a very different kind of ship. The admin knows everybody’s business before they’re allowed on the box, eh?
August 2nd, 2006 at 3:09 pm
Egad …. has the world gone mad?!?!?! Knowing everyone’s business is a risky affair and is better left for the politicians. It’s better to set early expectations that misbehavior results in immediately degraded QOS. So, if a sysadmin is going to do a good job she/he has to be willing to enforce some basic, behavioral rules. Rules like … “If you get caught spamming you’ll find yourself set with a 64k or smaller QOS until the spam complaints stop.” As I see it, spammers, or any miscreant for that matter, behave the way they do because the perceived pain related so said behavior is less than what they understand as reward. The trick is to make the realized pain (in this case degraded QOS to the spammer) higher than the spammer’s ability to achieve reward. For instance, a child sees a paperclip and an electrical wall socket. The first time this combination is realized the child thinks “hey, inserting Mr. Paperclip into Mr. Wall socket might make a cool spark.” Thus, the perceived reward is higher than the unknown pain. The first time the child sticks said paper clip into said electrical socket, their perception of the reward of seeing the magical spark wanes and is replaced by the realization that electricity is for electrical engines and light bulbs and NOT for human consumption.
August 2nd, 2006 at 3:33 pm
You know Not-Not, I think you’ve hit the problem on the head.
The anti-spam community figured out how to teach sys-admins the pain of allowing mail spammers on their services when they figured out they could use blocklists.
But so far we don’t have anything to punish providers who sell services to webspammers.
If we can figure out something that really hurts, but that isn’t seen as unfair, we might actually get somewhere.
Hostgator still hasn’t figured out how to stop the referrer spamming script. Frankly, I don’t think they even tried. It cost too much.
So, in terms of providers, the pain would have to be: It costs more to keep the spammers or the compromised script than to fix it.
When we solve that one, we can get somewhere with webspam.