[My correspondent's system had been broken into.] From rick Mon Jul 29 13:18:32 2002 Date: Mon, 29 Jul 2002 13:18:32 -0700 To: plug@lists.q-linux.com Subject: Re: [plug] suspicious files User-Agent: Mutt/1.3.28i Quoting manny@iconn.com.ph (manny@iconn.com.ph): > What should I do now? That's the bad news: You need to rebuild. Most important priorities: 1. Safeguard data. 2. Shut out the intruders. Then: 3. Keep them from coming back. 4. Notify appropriate persons, if any. The process of doing this is painful, so many admins try going halfway. That is almost always a bad idea, because the bad guys leave backdoor ways of getting back in, and you probably won't find them all (and, when they return, are often vindictive). The point is that you don't want to have to go through this pain _twice_, because you weren't careful enough the first time -- nor do you want your system wiped. No executables or configuration files on the machine can be trusted and should be assumed unusable. This includes /bin, /sbin, /usr/bin, /usr/sbin, /usr/X11R6/bin, /lib, /usr/lib, /etc, crontabs, ~/bin, and so on. At the moment, you are not in control of the machine, and can't predict when it might auto-erase or start attacking other sites elsewhere. Therefore, many would recommend flipping the AC power off, as your first step -- without an orderly shutdown. Now, without running _any_ code on the system's hard drive, bring up some other maintenance-type Linux system that can mount the server's partitions (e.g., a second hard drive, a maintenance floppy, an LNX-BBC CDR disk, etc.). Make at least one complete data backup of the server's files. Remove those backups and put them somewhere else for safekeeping. Read this thoroughly: http://www.cert.org/tech_tips/root_compromise.html Now, build a new Linux installation while on a private network, using no executables or configuration files from the old machine. Copy over the old machine's data files. Manually edit /etc/* to recreate the old machine's configuration, referring to the old configuration files but not trusting them. Make sure you examine users' home directories and remove any executables stored there. Recreate all user accounts, issuing all users without exception new passwords, but with their login temporarily disabled pending rebuild completion. Apply all needed security updates to your distribution. Look carefully at publicly accessible services, to ensure that none are enabled that you don't need, and that all remaining ones satisfy your local security policy. Consider installing logcheck and tripwire. Review your system configuration. Probe your machine using nmap. Connect your new server to the network. Probe the machine and nearby hosts again using nmap. Inform your users of their new passwords in person or via telephone, only (never e-mail). Sternly warn them that they must never re-adopt their former passwords -- if you trust them that far. If you don't, disable users' ability to change passwords. Enable user login. And re-read http://www.cert.org/tech_tips/root_compromise.html , just to make sure you didn't miss anything. -- Cheers, Rick Moen "vi is my shepherd; I shall not font." rick@linuxmafia.com -- Psalm 0.1 beta From rick Mon Jul 29 16:02:45 2002 Date: Mon, 29 Jul 2002 16:02:45 -0700 To: plug@lists.q-linux.com Subject: Re: [plug] suspicious files User-Agent: Mutt/1.3.28i Quoting manny@iconn.com.ph (manny@iconn.com.ph): > Actually, it may not be that bad. My machine is standalone and I'm the > only user. I'm thinking of doing a clean wipe, and resinstalling from the > CD. Of course, that would mean reinstalling all my applications all over > again. Could I still use the RPMs and tarballs I've downloaded, or is it > likely that these are compromised too? Are you feelin' lucky? ;-> Me, I'd grab the copies on my backups from before the security compromise. > As for my data, I can back them up and put them on removable media or > upload them somewhere. This might help: http://linuxmafia.com/~rick/linux-info/kb/200 -- Cheers, "It ain't so much the things we don't know that get us Rick Moen in trouble. It's the things we know that ain't so." rick@linuxmafia.com -- Artemus Ward (1834-67), U.S. journalist From rick Mon Jul 29 13:45:15 2002 Date: Mon, 29 Jul 2002 13:45:15 -0700 To: plug@lists.q-linux.com Subject: Re: [plug] suspicious files User-Agent: Mutt/1.3.28i Quoting manny@iconn.com.ph (manny@iconn.com.ph): > What does one do now? Is it simply a matter of deleting these files? The likely result of doing that is that your machine will continue to be compromised. Your data may disappear, be altered, or be sent to unauthorised parties. Your system may be used to carry out attacks on other systems (which will appear to be carried out by you). Not good. A traditional plan for recovery is outlined in my post of a few minutes ago. -- Cheers, My pid is Inigo Montoya. You kill -9 Rick Moen my parent process. Prepare to vi. rick@linuxmafia.com From rick Fri Oct 11 11:57:48 2002 Date: Fri, 11 Oct 2002 11:57:48 -0700 To: PLUG Subject: Re: [plug] emergency procedures with a cracked box User-Agent: Mutt/1.4i Quoting Gerald Timothy Quimpo (gquimpo@sni-inc.com): > http://online.securityfocus.com/news/1113 > > the guy whose cracked box the sendmail trojan was reporting to decided > to disallow access to port 6667 so that hacked sendmail would stop > reporting to him but he didn't disconnect his box from the network. > > apparently that tipped off the cracker and he lost years of data > because the cracker went through his box deleting stuff willy-nilly. > maybe for revenge, maybe to cover his tracks. Note that this experience would have been much, much, much less traumatic if the admin had current, tested backups. All he would have lost is a small amount of current machine state, the time required to rebuild the box, and the opportunity to do security forensics, to figure out how break-in occurred and how to prevent recurrence. Current _tested_ backups can save your... job, at least. "Tested" means you do occasional test restores onto a sacrificial machine, and verify that everything's there. Otherwise, how do you really know that your backups are sufficient? -- Cheers, The genius of you Americans is that you never make Rick Moen clear-cut stupid moves, only complicated stupid moves rick@linuxmafia.com that make us wonder at the possibility that there may be something to them that we are missing. --Gamel Abdel Nasser