Find details of runaway user

I had the great joy of having to troubleshoot a compromised server. The server fell victim to a security flaw in the software it ran, resulting in the process running malware.

Once we pulled the data off and decided to rebuild the box just to be sure, I wanted to quickly investigate what the compromised user was trying to do. So, started it up in a bubble, and wrote a quick script to watch the user work.

What it did, was fire off quick processes that dies quickly, making tracing the malware quite hard.

I replaced the compromised user with my name to show the output, here the script:

#!/bin/bash
USER=werner

PROCS=`ps -ef | grep ${USER} | grep -v grep | awk '{print $2}'`

if [[ ! -z "$PROCS" ]]; then
  echo "##########################################"
  echo "###### $(date) #####"
  echo "##########################################"
  for i in $PROCS; do
    # List processes
    echo "Processlist for $i: "
    #ps -ef | grep $i | grep -v grep
    ps u -U $USER -u $USER  --forest -p $i
    echo
    # List commandline
    echo "Commandline for $i: "
    cat /proc/$i/cmdline
    echo
    echo
    # List open files
    echo "Files used by $i: "
    ls -l /proc/$i/fd
    echo
    # List working dir
    echo "Command's working dir: "
    pwdx $i
    echo
    # Kill off the process
    if kill -9 $i; then
      echo "Kill Succeeded"
    else
      echo "Kill Failed"
    fi
    echo
    echo "--------------------------------------------------------"
    echo
  done
fi

The output of the script:

##########################################
###### Sat Nov  2 13:36:47 NZDT 2019 #####
##########################################
Processlist for 20729:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     20729  0.0  0.0 191932  2452 pts/1    S+   13:36   0:00 su - werner -c top
werner   20730  0.2  0.0 162156  2412 ?        Ss   13:36   0:00  \_ top

Commandline for 20729:
su-werner-ctop

Files used by 20729:
total 0
lrwx------. 1 root root 64 Nov  2 13:36 0 -> /dev/pts/1
lrwx------. 1 root root 64 Nov  2 13:36 1 -> /dev/pts/1
lrwx------. 1 root root 64 Nov  2 13:36 2 -> /dev/pts/1
lrwx------. 1 root root 64 Nov  2 13:36 3 -> socket:[13342951]
lrwx------. 1 root root 64 Nov  2 13:36 4 -> socket:[13345961]
l-wx------. 1 root root 64 Nov  2 13:36 6 -> /run/systemd/sessions/526.ref

Command's working dir:
20729: /

Kill Succeeded

--------------------------------------------------------

Processlist for 20730:
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
werner   20730  0.2  0.0 162156  2412 ?        Ss   13:36   0:00 top

Commandline for 20730:
top

Files used by 20730:
total 0
lrwx------. 1 werner werner 64 Nov  2 13:36 0 -> /dev/pts/1
lrwx------. 1 werner werner 64 Nov  2 13:36 1 -> /dev/pts/1
l-wx------. 1 werner werner 64 Nov  2 13:36 2 -> /dev/null
lrwx------. 1 werner werner 64 Nov  2 13:36 3 -> /dev/pts/1
lr-x------. 1 werner werner 64 Nov  2 13:36 4 -> /proc/stat
lr-x------. 1 werner werner 64 Nov  2 13:36 5 -> /proc/uptime
lr-x------. 1 werner werner 64 Nov  2 13:36 6 -> /proc/meminfo
lr-x------. 1 werner werner 64 Nov  2 13:36 7 -> /proc/loadavg

Command's working dir:
20730: /home/werner

Kill Succeeded

--------------------------------------------------------