Neil Turner's Blog

Blogging about technology and randomness since 2002

Keeping the server steady

Since August, I’ve been having problems with the performance of this server. It was slow, and was regularly getting overloaded. In some cases, a hard reboot was necessary to get the server up and running again.

Recently things have been better, following some minor tweaks, but the act of posting a new blog entry would still bring the site to its knees for up to half an hour afterwards.

Anyhow, about 24 hours ago I made some further changes. And – so far! – everything seems to be working fine now. Even posting last night’s rant didn’t affect performance. So here’s what I did to alleviate the issues that I was having:

1. Slim down WordPress by removing plugins

I got rid of three or four plugins that I deemed not strictly necessary. In some cases, some of the larger plugins like Jetpack and WordPress SEO updated to include the features of some simpler plugins. I now have just 10 plugins installed and enabled.

2. Block rogue crawler bots

The top 3 IP addresses that access this site were from the same IP range in the Netherlands, and used by a bot called ‘Owlinbot’, which was crawling my site every three seconds, day in, day out. So, I added lines to Apache’s .htaccess file to block both the IP addresses and the Owlinbot user agent. It’s still trying to crawl the site but it gets served a 403 HTTP error.

I also blocked a couple of others but Bad Behaviour generally does a good job of keeping badly behaved crawlers out.

3. Optimise Apache’s prefork MPM

I changed the default settings for Apache’s ‘prefork’ multi-processing modules (MPM), using some sample settings posted on ServerFault. As the article states, they are not in any way ‘magic numbers’ but the situation described is similar to mine, and they seem to work well.

4. Reduce Apache’s KeepAlive timeout

Apache defaults to allowing up to 15 seconds between requests, when a user agent uses ‘KeepAlive’ to keep the connection active. The ServerFault article suggests knocking this down to 2 seconds – I kept it at 5, but either way 15 seconds is probably over-generous.

5. Disable InnoDB in MySQL

All of my MySQL tables use the standard MyISAM database format, so I turned off InnoDB support as it’s not necessary and would free up memory. Again, this was listed in the ServerFault discussion linked above.

6. Disable POP3 access in Dovecot

Dovecot is the mail transfer agent that runs on my server, and by default it runs separate processes for IMAP and POP3 logins. I don’t use POP3, so disabling it frees up memory as the POP3 processes won’t run.

Doing all of these things has reduced the amount of memory used substantially. Although almost all of the RAM on the server is accounted for during normal operation, it now very rarely needs to resort to using virtual memory. And general web browsing performance is at least as good as before, if not better in some cases.

Hopefully this means that the problem is now sorted. It’s been rather frustrating over the past few months and is part of the reason why I haven’t been able to blog as much.


  1. We’ve started rolling out CloudLinux on our servers – it’s helped quite a bit on stablising things (we may be able to give you a discount if you decide to go for it – if you are running RedHat/CentOS, you’ll be able to install/uninstall at will). You could also look at adding Varnish, APC and/or Memcache(d) to the server – and don’t forget to tune mysql (see )

  2. It’s strange, from time to time, posting a new blog article will completely take down our company website. It’s usually MySQL that needs a kick, but we’ve never gotten to the bottom of it.

  3. Also, as above, APC will help out, Varnish will probably cut your server response time by a factor of 10. Not sure memcached will help out much. If you go down the Varnish route, make sure you get a decent plugin to burn the cache when you make new posts.