Minimizing server downtime

Suppose that you are responsible for a typical LAMP server running Debian. The goal is to have the needed security updates and minimize downtime. Unfortunately, due to installation of security updates, services need to be restarted, and sometimes (after kernel updates) even a reboot is required. Here are some tips that will help you shrink the time of service unavailability.

1. Upgrade your key services (apache, php and mysql) separately from the rest of software. Reason: if you just run "apt-get dist-upgrade" to upgrade everything, then your MySQL server will be stopped at the beginning of the whole upgrade and started only at the end, after unpacking all other packages. So, run "apt-get install mysql-server" before "apt-get dist-upgrade" to minimize the delay.
2. If you know that kexec works on your hardware, use it: apt-get install kexec-tools. This way, you will avoid the long time (sometimes more than a minute) while your server shows useless BIOS screens. Instead of rebooting to the BIOS, the old kernel will load the new kernel directly and pass control to it, thus avoiding the delay.

DNS servers in resolv.conf

Yesterday I found a problem with my Linode VPS. When running "aptitude update", it complained about not being able to resolve the name for security.debian.org (a server from where Debian distributes security updates).

Here is the /etc/resolv.conf file from that VPS, generated by dhclient:

domain members.linode.com
search members.linode.com
nameserver 109.74.194.20
nameserver 109.74.192.20
nameserver 109.74.193.20

It looks like the 109.74.194.20 server is overloaded and returns a SERVFAIL ("server failure - The name server was unable to process this query due to a problem with the name server") answer from time to time. Unfortunately, Debian has a bug in its version of libc that prevents the resolver from trying the other (working) servers if the first one returns an answer indicating a temporary failure.

The bug does not exist in glibc-2.12.1 on my Gentoo box at home. However, if the server does not respond at all, the Gentoo resolver spends 15 seconds before trying the next one, which is also not nice.

To work around the bug, you can run your own DNS server on 127.0.0.1 that forwards all queries to the official servers, and put 127.0.0.1 as the only nameserver in /etc/resolv.conf.