Neil Turner's Blog

Blogging about technology and randomness since 2002

Recovering from downtime

Screenshot of the top command

If you tried to view this site last week and couldn’t, then I apologise. Up until the weekend, this web server was frequently unreachable, and I’ve lost track of the number of times that I had to restart Apache, and/or the whole server. Thankfully, things seem to be better now.

I’m still not sure what caused it; whether I got hit by an overzealous web crawler, or a package update last weekend broke something, but either way the virtual machine that the server runs on was constantly running out of memory. Even when I tried to reign in Apache’s multiple processes, pages wouldn’t load.

Eventually on Thursday night I downloaded all of my data off the server onto my Mac, and installed a fresh virtual machine image. The previous image has been in use since I switched to Bytemark as a host a few years ago, and has seen an upgrade from Debian Lenny to Squeeze, and from Bytemark’s old vhost manager to their new Symbiosis manager. This fresh image meant that I didn’t have any pre-upgrade baggage, and a clean set of configuration files.

Even that didn’t initially solve the problem initially. It was only after I removed the W3 Total Cache plugin from WordPress that things got better. I’m not sure if it was something to do with how I configured the plugin, but, ironically, not havingW3 Total Cache running resulted in a bigger performance boost. Crucially, there were no out of memory errors.

Rebuilding the site has meant that some things aren’t working as yet. I haven’t got around to re-enabling SpamAssassin so I’m getting quite a bit more junk in my inbox at present (a lot of it in Portuguese for some reason…) and there are some other missing WordPress plugins that I haven’t reinstalled. I’m trying out WP Fast Cache as a more basic caching solution for the most frequently-visited pages, and it seems to work okay.

One Comment

  1. Stick newrelic.com on the box (the free version should be enough) which should give you an idea what is going on. It may also be worth running mysqltuner.pl to ensure MySQL is reasonably tuned.

    Server admin is one of those things where it’s easy to get the basics right, but once things start going “odd” is where it gets difficult… Apache can be tuned, email systems can be tuned, firewall systems etc (it’s probably worth installing the free “csf” firewall from ConfigServer as that has some basic DoS protection built in and is quite simple to use). It’s also worth considering CloudFlare if you think your traffic is getting too much for your host [n.b. I work for a company which is a CloudFlare hosting partner]

    Feel free to drop me a note if you are having problems