I'm an avid reader of TFL and missed it sorely when it went offline.
I'm also a senior Unix Sys Admin, and have been responsible for security at big and small web sites and have had my machines auditted by professional "Cybersecurity" teams. I don't want to be an armchair quarterback, but I'd like to offer a few suggestions. It sounded like you are running Linux, so I'll toss in some quickies:
1) If you are using a 2.2 series kernel, build a custom kernel using the openwall patches, and turn off loadable kernel modules. This should protect you against a majority of stack smashing exploits, and should someone break int o the machine via another means, will cut down on a bunch of ways hackers use to hide their tracks.
2) If you are using a 2.4 series kernel, get the grsecurity patches and configure as tight as you see fit. The recent 2.4 kernels are supposed to perform better under load than the 2.2 series.
3) Run something like Bastille Linux to turn off unnecessary setuid/setgid executables.
4) Run chkrootkit periodically to see if your system has been owned by some script kiddie. Make sure that chkrootkit's dependencies are isolated from the rest of the system. It may be a good idea to stabilize your machine, and build a tripwire database, and then install that and chkrootkit on a cdrom that you leave in the drive at all times.
5) Turn off all unnecessary services. Getting the latest patched versions of server code isn't such an issue if you aren't running the service to begin with. On production web servers, I usually turn off xinetd entirely, and only have 2 services: http and ssh. A stock RH Linux install sitting on the net only lasts a few hours before it gets owned.
6) If you have dependent servers, like mysqld, either have the client use IPC connections, or bind the server to localhost, and have all client connections go to the localhost address, so that nobody on the outside can even see the service is running.
7) Disaster recovery plan. Gotta have this if you want to have "defense in depth".
Most of these changes won't require any major changes in your application, and should tighten up security a lot. There are lots of other changes that could be made, but these are the high percentage ones that should keep out your basic script kiddie.
If the black hats got in via the heap overflow exploit listed for PHP, then the stack smashing defenses wouldn't have helped, but the intrusion detection tools would have detected them. You might want to look into running the web server chrooted, or migrate to the Java servlet based PHP engine. Java is safe against most buffer overflow attacks.
Anyway, thanks for all your hard work bringing the machine back up - we all appreciate it. If you need some extra help on the systems side, let me know, I wouldn't mind volunteering some of my time.
Steve
syc@onebox.com
I'm also a senior Unix Sys Admin, and have been responsible for security at big and small web sites and have had my machines auditted by professional "Cybersecurity" teams. I don't want to be an armchair quarterback, but I'd like to offer a few suggestions. It sounded like you are running Linux, so I'll toss in some quickies:
1) If you are using a 2.2 series kernel, build a custom kernel using the openwall patches, and turn off loadable kernel modules. This should protect you against a majority of stack smashing exploits, and should someone break int o the machine via another means, will cut down on a bunch of ways hackers use to hide their tracks.
2) If you are using a 2.4 series kernel, get the grsecurity patches and configure as tight as you see fit. The recent 2.4 kernels are supposed to perform better under load than the 2.2 series.
3) Run something like Bastille Linux to turn off unnecessary setuid/setgid executables.
4) Run chkrootkit periodically to see if your system has been owned by some script kiddie. Make sure that chkrootkit's dependencies are isolated from the rest of the system. It may be a good idea to stabilize your machine, and build a tripwire database, and then install that and chkrootkit on a cdrom that you leave in the drive at all times.
5) Turn off all unnecessary services. Getting the latest patched versions of server code isn't such an issue if you aren't running the service to begin with. On production web servers, I usually turn off xinetd entirely, and only have 2 services: http and ssh. A stock RH Linux install sitting on the net only lasts a few hours before it gets owned.
6) If you have dependent servers, like mysqld, either have the client use IPC connections, or bind the server to localhost, and have all client connections go to the localhost address, so that nobody on the outside can even see the service is running.
7) Disaster recovery plan. Gotta have this if you want to have "defense in depth".
Most of these changes won't require any major changes in your application, and should tighten up security a lot. There are lots of other changes that could be made, but these are the high percentage ones that should keep out your basic script kiddie.
If the black hats got in via the heap overflow exploit listed for PHP, then the stack smashing defenses wouldn't have helped, but the intrusion detection tools would have detected them. You might want to look into running the web server chrooted, or migrate to the Java servlet based PHP engine. Java is safe against most buffer overflow attacks.
Anyway, thanks for all your hard work bringing the machine back up - we all appreciate it. If you need some extra help on the systems side, let me know, I wouldn't mind volunteering some of my time.
Steve
syc@onebox.com
