A machine of a friend of mine got cracked, this is an analysis of the attack.

Everything began on a wonderful Monday morning...

soohrt@sit0:~$ ssh 62.208.xxx.xxx
soohrt@62.208.xxx.xxx's password:
Permission denied, please try again.

[ It's too early to type, so again ]

soohrt@62.208.xxx.xxx's password:
Permission denied, please try again.

[ *yawn*, damn ]

soohrt@62.208.xxx.xxx's password:
Permission denied (publickey,password,keyboard-interactive).

[ I can't be that stupid, there has to be something botched. ]

I suspected that there was something wrong, usually I get my passwort right the first time. ;)
So I checked my second account

soohrt@sit0:~$ ssh heureka@62.208.xxx.xxx
heureka@62.208.xxx.xxx's password:

Good news everyone.
Last login: Tue May 21 00:23:52 2002 from p5081b022.dip.t-dialin.net
heureka@claudius07:~$

Ok, it works. That's a start. Now I want to know, why I can't login as 'soohrt'.

heureka@claudius07:~$ id soohrt
id: soohrt: No such user

[ WTF? ]

heureka@claudius07:~$ su -
Password:
root@claudius07:~# id soohrt
id: soohrt: No such user
root@claudius07:~# grep hrt /etc/passwd
sohrt:x:517:100::/home/sohrt:/bin/bash

[ There's something serious going on, accountnames don't change overnight. ;) ]

root@claudius07:~# ls /home/*hrt
ls: /home/*hrt: No such file or directory

Both my homedir and the fake accounts $HOME weren't there.
ps/top didn't show unusual activity, lets try last

root@claudius07:~# last|head -n7

root	pts/6	p5081b022.dip.t-	Mon Sep 23 20:41 still logged in
root	pts/5	p5088fb85.dip.t-	Mon Sep 23 20:28 still logged in
root	pts/0	p50916788.dip.t-	Mon Sep 23 20:17 - 21:11 (00:53)
root	pts/3	p5081b022.dip.t-	Mon Sep 23 20:13 still logged in
root	pts/0	p50916788.dip.t-	Mon Sep 23 20:13 - 20:13 (00:00)
sohrt	pts/0	ws01.host.netent	Mon Sep 23 01:55 - 02:02 (00:07)
soohrt	pts/0	pd9e30308.dip.t-	Sun Sep 22 22:14 - 22:14 (00:00)

What the fuck is going on here? The session at 9/22 22:14 is ok, I was checking emails, but what happened after that?

root@claudius07:~# chkrootkit
-su: chkrootkit: command not found
[ I remember installing and using chkrootkit, so obviously someone/thing doesn't want to be found. ]

I grabbed a clean chkrootkit from my local debian installation and transferred it to the server

root@claudius07:~# ./chkrootkit
ROOTDIR is `/'
Checking `amd'... not found
Checking `basename'... not infected
Checking `biff'... not found
Checking `chfn'... not infected
Checking `chsh'... not infected
Checking `cron'... not infected
Checking `date'... not infected
Checking `du'... not infected
Checking `dirname'... not infected
Checking `echo'... not infected
Checking `egrep'... not infected
Checking `env'... not infected
Checking `find'... not infected
Checking `fingerd'... not found
Checking `gpm'... not found
Checking `grep'... not infected
Checking `hdparm'... not infected
Checking `su'... not infected
Checking `ifconfig'... not infected
Checking `inetd'... {idedisk_init} {pci_set_power_state}
Warning: /boot/System.map does not match kernel data.
not infected
Checking `inetdconf'... not infected
Checking `identd'... not found
Checking `killall'... not infected
Checking `ldsopreload'... not infected
Checking `login'... not infected
Checking `ls'... not infected
Checking `lsof'... not infected
Checking `mail'... not infected
Checking `mingetty'... not found
Checking `netstat'... not infected
Checking `named'... not found
Checking `passwd'... not infected
Checking `pidof'... not infected
Checking `pop2'... not found
Checking `pop3'... not found
Checking `ps'... not infected
Checking `pstree'... not infected
Checking `rpcinfo'... not infected
Checking `rlogind'... not found
Checking `rshd'... not found
Checking `slogin'... not infected
Checking `sendmail'... not infected
Checking `sshd'... {idedisk_init} {pci_set_power_state}
Warning: /boot/System.map does not match kernel data.
not infected
Checking `syslogd'... not infected
Checking `tar'... not infected
Checking `tcpd'... {idedisk_init} {pci_set_power_state}
Warning: /boot/System.map does not match kernel data.
not infected
Checking `top'... not infected
Checking `telnetd'... not infected
Checking `timed'... not found
Checking `traceroute'... not infected
Checking `w'... not infected
Checking `write'... not infected
Checking `aliens'... no suspect files
Searching for sniffer's logs, it may take a while... nothing found
Searching for HiDrootkit's default dir... nothing found
Searching for t0rn's default files and dirs... nothing found
Searching for t0rn's v8 defaults... nothing found
Searching for Lion Worm default files and dirs... nothing found
Searching for RSHA's default files and dir... nothing found
Searching for RH-Sharpe's default files... nothing found
Searching for Ambient's rootkit (ark) default files and dirs... nothing found
Searching for suspicious files and dirs, it may take a while... nothing found
Searching for LPD Worm files and dirs... nothing found
Searching for Ramen Worm files and dirs... nothing found
Searching for Maniac files and dirs... nothing found
Searching for RK17 files and dirs... nothing found
Searching for Ducoci rootkit... nothing found
Searching for Adore Worm... nothing found
Searching for ShitC Worm... nothing found
Searching for Omega Worm... nothing found
Searching for Sadmind/IIS Worm... nothing found
Searching for MonKit... nothing found
Searching for Showtee... nothing found
Searching for OpticKit... nothing found
Searching for T.R.K... nothing found
Searching for Mithra... nothing found
Searching for anomalies in shell history files... nothing found
Checking `asp'... not infected
Checking `bindshell'... INFECTED (PORTS: 1008)
Checking `lkm'... You have 1 process hidden for readdir command
You have 1 process hidden for ps command
Warning: Possible LKM Trojan installed
Warning: /boot/System.map does not match kernel data.
Checking `rexedcs'... not found
Checking `sniffer'... eth0 is not promisc eth0:0 is not promisc eth0:1 is not promisc eth0:2 is not promisc eth0:3 is not promisc
Checking `wted'... nothing deleted
Checking `z2'...

Ok, that clears it up a little. This server is definitely compromised. The bindshell alert is ok, thats normal on this machine.
Since ps/top/pstree didn't show anything and chkrootkit reports a hidden process, I tried to read directly from /proc/

root@claudius07:/proc# for i in *; do test -f $i/cmdline && (cat $i/cmdline; echo $i); done
init.1
2
3
4
5
6
/usr/bin/perl./usr/sbin/popauther3.pl.144
/usr/sbin/sshd.192
/sbin/syslogd.201
/sbin/klogd.-c.1.207
/usr/sbin/atd.223
/usr/sbin/mysqld.--user=mysql.--pid-file=/var/lib/mysql/mysqld.pid.--datadir=/var/lib/mysql.263
/usr/sbin/mysqld.--user=mysql.--pid-file=/var/lib/mysql/mysqld.pid.--datadir=/var/lib/mysql.267
/usr/sbin/mysqld.--user=mysql.--pid-file=/var/lib/mysql/mysqld.pid.--datadir=/var/lib/mysql.268
/usr/sbin/cron.364
/usr/sbin/nscd.376
/usr/sbin/nscd.377
/usr/sbin/nscd.378
/usr/sbin/nscd.379
/usr/sbin/nscd.380
/usr/sbin/nscd.381
/usr/sbin/nscd.382
/usr/sbin/inetd.400
/usr/sbin/httpd.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.420
/usr/bin/perl./usr/libexec/webmin/miniserv.pl./etc/webmin/miniserv.conf.439
/sbin/mingetty.--noclear.tty1.442
/sbin/mingetty.tty2.443
/sbin/mingetty.tty3.444
/sbin/mingetty.tty4.445
/sbin/mingetty.tty5.446
/sbin/mingetty.tty6.447
/sbin/getty.ttyS0.DT9600.vt100.448
-bash.1556
/usr/sbin/named.2072
/usr/sbin/named.2073
/usr/sbin/named.2074
/usr/sbin/named.2075
/usr/sbin/named.2076
/usr/sbin/named.2077
sendmail: accepting connections.4050
/usr/bin/perl./usr/sbin/popauther3.pl.4054
./psybnc.5942
/usr/sbin/httpd.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.6750
/usr/sbin/httpd.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.7388
SCREEN.BitchX.irc.fu-berlin.de.-H.62.208.xxx.xxx.11440
BitchX.irc.fu-berlin.de.-H.62.208.xxx.xxx.11441
tail.-f./var/log/allmessages.11820
proftpd (accepting connections)12836
/usr/sbin/sshd.18076
-bash.18077
/usr/sbin/sshd.19737
-bash.19738
/usr/sbin/sshd.20731
-bash.20732
/bin/sh./usr/bin/updatedb.23856
/bin/sh./usr/bin/updatedb.23864
sort.-f.23865
/usr/lib/find/frcode.23866
/usr/bin/find./.(.-fstype.nfs.-o.-fstype.NFS.-o.-fstype.proc.-o.-fstype.afs.-o.-fstype.smbfs.-o.-fstype.autofs.-o.-type.d.-regex.\(^/tmp$\)\|\
/usr/sbin/httpd.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.27581
/usr/lib/java//bin/../bin/i686/green_threads/java.org/apache/jserv/JServ./etc/httpd/jserv/jserv.properties.27582
/usr/sbin/fcgi-.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.27587
/usr/bin/perl./root/confixx/pipelog.pl.27593
/usr/sbin/sshd.27743
./eggdrop.BKA.28552

Doesn't look suspicious, but since chkrootkit reported that readdir returns compromised results, I tried something else

root@claudius07:/proc# for i in `seq 1 33000`; do test -f $i/cmdline && (cat $i/cmdline; echo $i); done
init.1
2
3
4
5
6
/usr/bin/perl./usr/sbin/popauther3.pl.144
/usr/sbin/sshd.192
/sbin/syslogd.201
/sbin/klogd.-c.1.207
/usr/sbin/atd.223
/usr/sbin/mysqld.--user=mysql.--pid-file=/var/lib/mysql/mysqld.pid.--datadir=/var/lib/mysql.263
/usr/sbin/mysqld.--user=mysql.--pid-file=/var/lib/mysql/mysqld.pid.--datadir=/var/lib/mysql.267
/usr/sbin/mysqld.--user=mysql.--pid-file=/var/lib/mysql/mysqld.pid.--datadir=/var/lib/mysql.268
/usr/sbin/cron.364
/usr/sbin/nscd.376
/usr/sbin/nscd.377
/usr/sbin/nscd.378
/usr/sbin/nscd.379
/usr/sbin/nscd.380
/usr/sbin/nscd.381
/usr/sbin/nscd.382
/usr/sbin/inetd.400
/usr/sbin/httpd.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.420
/usr/bin/perl./usr/libexec/webmin/miniserv.pl./etc/webmin/miniserv.conf.439
/sbin/mingetty.--noclear.tty1.442
/sbin/mingetty.tty2.443
/sbin/mingetty.tty3.444
/sbin/mingetty.tty4.445
/sbin/mingetty.tty5.446
/sbin/mingetty.tty6.447
/sbin/getty.ttyS0.DT9600.vt100.448
-bash.1556
/usr/sbin/named.2072
/usr/sbin/named.2073
/usr/sbin/named.2074
/usr/sbin/named.2075
/usr/sbin/named.2076
/usr/sbin/named.2077
sendmail: accepting connections.4050
/usr/bin/perl./usr/sbin/popauther3.pl.4054
./psybnc.5942
/usr/sbin/httpd.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.6750
/usr/sbin/httpd.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.7388
SCREEN.BitchX.irc.fu-berlin.de.-H.62.208.xxx.xxx.11440
BitchX.irc.fu-berlin.de.-H.62.208.xxx.xxx.11441
tail.-f./var/log/allmessages.11820
proftpd (accepting connections)12836
./sk.15252
/usr/sbin/sshd.18076
-bash.18077
/usr/sbin/sshd.19737
-bash.19738
/usr/sbin/sshd.20731
-bash.20732
/usr/sbin/httpd.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.27581
/usr/lib/java//bin/../bin/i686/green_threads/java.org/apache/jserv/JServ./etc/httpd/jserv/jserv.properties.27582
/usr/sbin/fcgi-.-f./etc/httpd/httpd.conf.-D.MODULES.-D.JSERV.-D.PERL.-D.PHP4.-D.SSL.27587
/usr/bin/perl./root/confixx/pipelog.pl.27593
/usr/sbin/sshd.27743
./eggdrop.BKA.28552

pid 15252 (./sk) looks interesting...

root@claudius07:/proc# ls 15252
ls: 15252: No such file or directory
root@claudius07:/proc# cd 15252
root@claudius07:/proc/15252#

[ gotcha! ]

root@claudius07:/proc/15252# ls -l

Since 'sk' was started from the current directory, environ should contain PWD

root@claudius07:/proc/15252# cat environ
PWD=/usr/share/locale/ro_US PAGER=less HOSTNAME=claudius07 RC_LANG=de_DE LS_OPTIONS=-a -N --color=tty -T 0 ignoreeof=0 POVRAYOPT=-l/usr/lib/povray/include SUSE_DOC_HOST=localhost LESSKEY=/etc/lesskey bib LESSOPEN=|lesspipe.sh %s.MANPATH=/usr/local/man:/usr/share/man:/usr/man:/usr/X11R6/man:/usr/openwin/man NNTPSERVER=news LESS=-M -S -I USER=root LS_COLORS= HISTCONTROL=ignoredups MACHTYPE=i386-suse-linux XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB RC_LC_COLLATE=POSIX GNOMEDIR=/opt/gnome COLORTERM=1 INFOPATH=/usr/local/info:/usr/share/info:/usr/info LOGNAME=root SHLVL=2 TEXINPUTS=:~/.TeX:/usr/share/doc/.TeX:/usr/doc/.TeX LC_CTYPE=de_DE MINICOM=-c on INFODIR=/usr/local/info:/usr/share/info:/usr/info SHELL=/bin/bash PRINTER=lp HOSTTYPE=i386 OSTYPE=linux WINDOWMANAGER=/usr/X11R6/bin/kde TERM=dumb HOME=/root XNLSPATH=/usr/X11R6/lib/X11/nl .no_proxy=localhost PATH=/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/usr/X11R6/bin:/bin:/usr/lib/java/bin:/usr/games/bin:/usr/games:/opt/gnome/bin LESSCHARSET=latin1 FROM_HEADER=noname.censored.de _=./sk OLDPWD=/usr/share/locale/ro_US

[ That looks promising ]

root@claudius07:/proc/15252# ls /usr/share/locale/ro_US
ls: /usr/share/locale/ro_US: No such file or directory

[ Again? ]

root@claudius07:/proc/15252# cd /usr/share/locale/ro_US
root@claudius07:/usr/share/locale/ro_US#

[ strike :) ]

root@claudius07:/usr/share/locale/ro_US# ls -lA

-rw--w--w-	root	root	1216 Sep 23 21:14 .sniffer
-rwxr-xr-x	root	root	29111 Sep 23 02:02 juno
-rwxr-xr-x	root	root	14916 Sep 23 02:02 sk

root@claudius07:/usr/share/locale/ro_US# strings sk
>ascii crap, but in the end:<
UPX!

[ So this file is packed with upx ]

root@claudius07:/usr/share/locale/ro_US# upx -d sk
Unpacked 1 file.
root@claudius07:/usr/share/locale/ro_US# strings sk
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:./bin:/usr/share/locale/ro_US:/usr/share/locale/ro_US/bin
HOME=/usr/share/locale/ro_US
HISTFILE=/dev/null
PS1=\[\033[1;30m\][\[\033[0;32m\]\u\[\033[1;32m\]@\[\033[0;32m\]\h \[\033[1;37m\]\W\[\033[1;30m\]]\[\033[0m\]#
SHELL=/bin/bash
TERM=linux
pqrstuvwxyzabcde
0123456789abcdef
/dev/ptmx
/dev/pty
/dev/tty
/dev/null
/dev/null
Can't open a tty, all in use ?
Can't fork subshell, there is no way...
/usr/share/locale/ro_US
/bin/sh
Can't execve shell!
BD_Init: Starting backdoor daemon...
FUCK: Can't allocate raw socket (%d)
FUCK: Can't fork child (%d)
Done, pid=%d
/usr/share/locale/ro_US/.rc
use:
%s <uivfp> [args]
u - uninstall
i - make pid invisible
v - make pid visible
f [0/1] - toggle file hiding
p [0/1] - toggle pid hiding
Detected version: %s
FUCK: Failed to uninstall (%d)
Suckit uninstalled sucesfully!
FUCK: Failed to hide pid %d (%d)
Pid %d is hidden now!
FUCK: Failed to unhide pid %d (%d)
Pid %d is visible now!
file
Failed to change %s hiding (%d)!
%s hiding is now %s!
kmalloc
_kmalloc
__kmalloc
/usr/share/locale/ro_US
/dev/kmem
FUCK: Can't open %s for read/write (%d)
RK_Init: idt=0x%08x,
FUCK: IDT table read failed (offset 0x%08x)
FUCK: Can't find sys_call_table[]
sct[]=0x%08x,
FUCK: Can't find kmalloc()!
kmalloc()=0x%08x, gfp=0x%x
FUCK: Can't read syscall %d addr
Z_Init: Allocating kernel-code memory...
FUCK: Out of kernel memory!
Done, %d bytes, base=0x%08x
/dev/kmem
ro_US
/dev/null
core
FUCK: Got signal %d while manipulating kernel!
/sbin/initro_US
0123456789abcdefghijklmnopqrstuvwxyz
0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ
NULL
/dev/null
1.3b
ro_US
/usr/share/locale/ro_US/.sniffer
/proc/
/proc/net/
socket:[
/sbin/init
/sbin/initro_US
login
telnet
rlogin
rexec
passwd
adduser
mysql
ssword:

Very nice, the rootkit tells us its name. A little googling shows the following page: http://www.phrack.com/show.php?p=58&a=7.

root@claudius07:/usr/share/locale/ro_US# ./sk u
/dev/null
Detected version: 1.3b
Suckit uninstalled sucesfully!
root@claudius07:/usr/share/locale/ro_US# pidof ./sk
15252
root@claudius07:/usr/share/locale/ro_US# kill -9 15252
root@claudius07:/usr/share/locale/ro_US# kill -9 15252
-su: kill: (15252) - No such process

After that, I grabbed the logs, and shut the machine down.
It's 3:24am and I'm tired, the continued post-mortem analysis will follow shortly.

That's all folks, bye for now

Update #1 (28.01.2003):

I got a couple of questions what I did after shutting the server down
We had a clean rescue-cd in the cdrom drive and so we were abled to boot a clean system (via the cheap "remote-hand" service from the data centre), restore a backup that was done a couple of days before the attack, upgrade the services and restart the server.
This is not the recommended way since you can't be absolutely sure that the backup was made before the attack! (well we were pretty sure, because the incremental backup logs showed no sign of changed files in /, and /usr for over 2 months) But still, there is a small amount of insecurity.