I’m a good programmer but no more than a so-so sysadmin. In this post, I’ll outline the stumbling steps I took to frustrate hacking attempts on my little server, the humble EC2 mini instance that is www.usefuljs.net. Self-hosting on Amazon’s computing infrastructure is a popular solution for gaining a web presence these days, so hopefully others will find the information here useful.

A few weeks ago I started noticing that my website was going down regularly – all I’d see was the message “Error establishing a database connection” when I visited it. It was kinda mysterious since I’d not changed anything on the server and it’s not exactly a heavy-use site. Anyway, the first couple of times, I simply restarted the database server, mysqld:

$ sudo su -
# /etc/init.d/mysqld restart

Unfortunately, this didn’t fix things for long and the site would soon go down again. Time to look in the logs:

$ sudo vi /var/log/mysqld.log

The relevant section from the log looks like this:

160125 10:50:52 mysqld_safe Number of processes running now: 0
160125 10:50:52 mysqld_safe mysqld restarted
160125 10:50:53 [Note] /usr/libexec/mysql55/mysqld (mysqld 5.5.45) starting as process 16301 ...
...
160125 10:50:53 InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
160125 10:50:53 InnoDB: Completed initialization of buffer pool
160125 10:50:53 InnoDB: Fatal error: cannot allocate memory for the buffer pool
160125 10:50:53 [ERROR] Plugin 'InnoDB' init function returned error.
160125 10:50:53 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
160125 10:50:53 [ERROR] Unknown/unsupported storage engine: InnoDB
160125 10:50:53 [ERROR] Aborting

In case it’s not obvious, here’s what’s happening: the mysqld monitor is noticing that MySQL has crashed (“Number of processes running now: 0”) and attempts to restart it. MySQL tries to allocate a big chunk of RAM (“Initializing buffer pool, size = 128.0M”), fails and gives up. As to why MySQL was falling over in the first place, it was a result of all the memory on my system getting used up. When this happens, the Linux kernel goes looking for processes to kill. Since database servers tend to use a fair amount of memory and since I don’t have a lot else running on my server, MySQL would consistently fall victim to the OOM-killer.

It seemed as though I needed more memory. EC2 mini instances have 2GB which is actually quite a lot when you consider that my first development workstation had 8MB (250 times smaller). I can’t add more physical RAM and I don’t want to pay for a larger instance. What I can do is add some swap. In case you don’t know what swap is, it means using the hard disk as backup memory. Memory that isn’t currently being accessed can be written out to disk so that another process can use it. When the original memory address is accessed again, the kernel will read the data from the disk, swapping another bit of memory out to disk in its place. An Amazon Linux installation doesn’t come with any swap enabled and let’s face it, enabling it is not something we’re born knowing how to do, so here’s how it’s done. First, we need to create the swap file itself:

# dd if=/dev/zero of=/var/swap.1 bs=1M count=1024

/dev/zero is a special file that gives you an infinite stream of zeroes when you read it; the above dd command reads 1024 one MB blocks from /dev/zero and writes them out to /var/swap.1. This creates a 1GB file full of zero bytes. The next step is to turn /var/swap.1 into an actual swap file and then tell the system to use it, like this:

# mkswap /var/swap.1
# swapon !$

To make our swap file persist between reboots, we need to add the following line to the end of /etc/fstab:

/var/swap.1 swap swap defaults 0 0

It turns out that this wasn’t the solution to my problems since the next day my website was down again. Nevertheless, swap is useful so I decided to keep it. My next suspicion was memory leaks. A memory leak occurs when a process grabs a chunk of memory and then forgets to give it back when it’s finished with it. I’m running WordPress which is written using PHP which itself runs in the Apache webserver so there’s at least three things that can leak memory. I am unable to get an answer other than “maybe” as to whether the WordPress / PHP combo is threadsafe which means that I’m using the inefficient, memory-hogging prefork Apache model. I created the following bit of Apache config to try to control memory use:

# cat /etc/httpd/conf.d/prefork.conf 
<IfModule mpm_prefork_module>
    StartServers 3
    MinSpareServers 3
    MaxSpareServers 6
    MaxClients 90
    MaxRequestsPerChild 2000
</IfModule>

The MaxRequestsPerChild causes Apache to kill its child processes when they have serviced 2000 requests. The Apache philosophy is that they’re never going to be able to eliminate all memory leaks so they can work around them by killing possibly leaky processes every so often since that will free up any memory they used. In case you’re wondering, this wasn’t the solution to my problems either.

The next time my site went down and I restarted MySQL and Apache, I noticed that the number of Apache processes quickly rose to about sixty. For a light-use site, that is ridiculous. It seemed as though I was under attack. The Apache access log showed what was going on:

# tail -f /etc/httpd/logs/access_log
185.130.5.209 - - [31/Jan/2016:19:29:09 +0000] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
...Repeated many times per second

The trouble with something like WordPress is that it’s an enormous, complicated piece of software that does loads of stuff that I don’t even know about. Such as providing an XML-RPC channel that can be used to attempt to brute-force account credentials so that some dickhead can vandalize my site with spam links. The access log showed that several IP addresses from the 185.130.5 subnet had been hammering my server with POSTs to xmlrpc.php. The first thing I wanted to do was find out who was attacking me. The whois utility will tell me who owns the subnet so that, for example, I can report abuse. Unfortunately, it wasn’t installed. The package that provides the utility is non-obviously called jwhois:

# yum install jwhois
# whois 185.130.5.209

From the output it seems like the Sindicate Group based in Lithuania has handed the subnet to some US organization called ohs4you.net who don’t even have a website. It didn’t look to me as though innocent individuals were unwittingly participating in distributed attacks. Criminals need ISPs too and it seems as though some ISPs in the more lawless parts of the world are happy to service them.

The next thing to do was block the swine. There are many ways one can go about this. The method that uses the least resources is to block them at the network level. The Linux kernel has a firewall module called iptables that can be configured to do many things. What I wanted to do was drop any connections from that subnet:

# iptables -I INPUT -s 185.130.5.0/255.255.255.0 -j DROP
# service iptables save

This is not sufficient, of course, because other botnets are going to try and attack my site and I want to stop them before they bring my site down. That wretched xmlrpc.php script is needed by some WordPress plugins (including one I installed, Jetpack). One can get yet more plugins that disable the offending RPC method calls but that isn’t satisfactory. Do I really want to invoke a PHP script to respond to requests from criminals? That is a computationally expensive way to say “no”. It’s better to get Apache itself to deny the request before it’s actually serviced. WordPress drops a .htaccess file in its install directory (on my system, that’s /var/www/html) that can be edited to control access to the site. I edited it to add the following entry:

<files xmlrpc.php>
order deny,allow
deny from all
#allow from 192.0.64.0/18
</files>

The order deny,allow line is somewhat unintuitive. The commented-out line is a subnet used by the Jetpack plugin to access xmlrpc.php. If I’d reversed things with order allow,deny, I couldn’t make exceptions to the deny rule even though it looks as though that should work. As to why a piece of software installed on my server requires IP addresses that I know nothing about to make inbound connections to my server to do whatever it is that it does (like the amazingly valuable service of putting notifications on Facebook when I publish a post) is another question entirely!

Now, I need to keep an eye on my access logs to see who’s probing me and then add them to the kernel block list. I’ll probably write a script to automate the process. In the meantime, I need to consider my choice of WordPress as a blogging platform. I’m very suspicious of code that opens up backdoors and side-channels without my knowledge or say-so. I’m thinking a better alternative might be something like Jekyll since a static site is pretty much immune from being hacked in the first place.