Root filesystem snapshots and kernel upgrades

On my laptop (which is running Arch), I decided to have periodic snapshots of the filesystem, in order to revert bad upgrades (especially those involving a large and unknown set of interdependent packages) easily. My toolset for this task is LVM2 and Snapper. Yes, I know that LVM2 is kind-of discouraged, and Snapper also supports btrfs, but most of the points below apply to btrfs, too.

Snapper, when used with LVM2, requires not just LVM2, but thinly-provisioned LVM2 volumes. Fortunately, Arch can have root filesystem on such volumes, so this is not a problem.

So, I have /boot on /dev/sda1, LVM on LUKS on /dev/sda2, root on a thinly-provisioned logical volume, and /home on another thinly-provisioned volume. And also swap on a non-thinly-provisioned volume. A separate /boot partition is needed because boot loaders generally don't understand thinly-provisioned LVM volumes, especially on encrypted disks. A separate volume for /home is needed because I don't want my new files in /home to be lost if I revert the system to its old snapshot. The same need to make a separate volume applies to other directories that contain data that should be preserved, but there are no such directories on my laptop. They can appear if I install e.g PostgreSQL.

And now there is a problem. Rollback to a snapshot works, but only if there were no kernel updates between the time when the snapshot was taken and when an attempt to revert was made. The root cause is that the kernel image is in /boot, and loadable modules for it are in /usr/lib/modules. The modules are reverted, but the boot loader still loads a new kernel, which now has no corresponding modules.

There are two solutions: either revert the kernel and its initramfs, too, when reverting the root file system, or make sure that modules are not reverted. I have not investigated how to make the first option possible, even though it would be a perfect solution. However, I have tried to make sure that modules are not reverted, and I am not satisfied with the result.

The idea was to move modules to /boot/modules, and make this location available somehow as /usr/lib/modules. Here "somehow" can mean either a symlink, or a bind mount. A symlink doesn't work, because the kernel upgrade in Arch will restore it back to a directory. A bind mount doesn't work, either. The issue is that, by putting modules on non-root filesystem, one creates a circular dependency between local filesystem mounting and udev (this would apply to a symlink, too).

Indeed, systemd-udevd, on startup, maps the /usr/lib/modules/`uname -r`/modules.alias.bin file into memory. So, now it has a (real) dependency on /usr/lib/modules being mounted. However, mounting local filesystems from /etc/fstab sometimes depends on systemd-udevd, because of device nodes. So, bind-mounting /usr/lib/modules merely from /etc/fstab, using built-in systemd tools, cannot work.

But it can work from a wrapper that starts before the real init:

#!/bin/sh
mount -n /boot              # /dev/sda1 is in devtmpfs and doesn't need udev
mount -n /usr/lib/modules   # there is still a line in fstab about that
exec /sbin/init "$@" 

But that's ugly. In the end, I removed the wrapper, installed an old known-working "linux" package, made a copy of the kernel, its initramfs and modules, upgraded the kernel again, and put the saved files back, so that they are now not controlled by the package manager. So now I have a known good kernel down in the boot menu, and knowledge that its modules will always be present in my root filesystem if I don't revert further than up to today's state.

And now one final remark. Remember that I said: "The same need to make a separate volume applies to other directories that contain data that should be preserved"? There is a temptation to apply this to the whole /var directory, but that would be wrong. If a system is being reverted to its old snapshot, a package database (which is in /var/lib/pacman) should be reverted, too. But /var/lib/pacman is under /var.

The conclusion is that Linux plumbers should think a bit about this "revert the whole system" use case, and maybe move some directories.

Still using icims.com for recruiting? Think again!

If your company has open vacancies and uses some system for pre-screening candidates (e.g. by giving them questions), I have a "small" task for you. Go to your system, answer the questions as if you were a candidate, validate the answers as you would expect from a candidate (e.g. actually perform the actions that the answer describes), and then save the results. Look at the whole process. Make a conclusion for yourself whether your system is usable for the stated purpose. Communicate it to your management, if needed.

If you are using icims.com for hiring technical candidates, the answer is most probably "not suitable at all".

The most annoying bug that icims.com has is that it does not allow the candidate to enter certain characters in certain positions. The exact error message is:

Q3 2 Contains invalid characters. You cannot use the characters: ' " \ / or ` in an enclosing instance of <>, <<, >> or ><.

 This triggers at least on the following types of input:

• XML or HTML
• Command redirections, e.g.: echo "foo bar" >> baz.txt
• Sequences of menu items to click, e.g.: "File > New > Folder", if a bad character happens to be before that

So, you cannot ask questions about HTML, shell scripting, or even general questions about using GUI-based applications.

This error message probably means that they are concerned about XSS attacks. However, filtering out invalid characters is a very sloppy way of protection against such attacks. And it imposes completely unreasonable restrictions on the user input.

In fact, any kind of input (including XML, bash scripts or text about clicking the menu) should be suitable, and can be made to display safely and properly in any browser, just by escaping the special characters when generating the HTML page. Many template engines exist that do this escaping for you automatically. Today, there is simply no reason not to use them.

If a candidate sees such error, he/she becomes demotivated. It is a stupid barrier before getting the correct answer to you. It also indicates that you don't care about your customers (by choosing business partners that allow such sloppy practices). Worse, some of your candidates (who see icims.com for the first time) can think that it is your product, or your internal system, and that you (not icims.com) have web developers with insufficient skills. I.e. that your company is not good enough to work in, because you don't weed out underqualified workers.

You don't want to lose candidates. So you don't want to use icims.com. Really.

Namespace issues in Apache

Some time ago I was given a task to write a Python WSGI script that handles all (really all!) URLs on a given Apache-based virtual host. Doesn't that sound easy? Here is the "obvious" part of the Apache configuration file for that virtual host:

WSGIScriptAlias / /path/to/script.wsgi

Except that it is slightly wrong. The problem is that the script really should handle all URLS, even "evil" ones, without any omissions. In the above form, it doesn't.

First, if the URL contains a percent-encoded slash (e.g., as in http://example.org/foo%2fbar), Apache gives a 404 error by default, without even calling the script. Solution:

AllowEncodedSlashes On

Wait, there is more! Regular aliases have higher priority than our WSGIScriptAlias, and that there are some default aliases usually set by distributions. Due to that, http://example.org/icons/folder.gif will map to a static file, not to the WSGI script. There is a wishlist bug in Apache that one cannot easily remove aliases from the namespace.

The solution (or, more precisely, a bad hack) that I found is to use mod_rewrite. By adding some prefix to all URLs (and, of course, dealing with it in the script), one makes sure that the existing aliases are not hit:

AllowEncodedSlashes On

RewriteEngine On
RewriteRule ^/(.*)$ /dummyprefix/$1 [PT]

WSGIScriptAlias /dummyprefix /path/to/script.wsgi

Still, this doesn't work (gives a 404 error instead of calling the script) on URLs that include some bad characters such as newlines, e.g. http://example.org/foo%0abar . Removing the dollar sign from the pattern fixes the 404 error, calls the script, but it then receives a truncated PATH_INFO in the environment. Still, SCRIPT_URI is correct, so this may be enough. In fact, if the script doesn't care about PATH_INFO, even this works:

AllowEncodedSlashes On

RewriteEngine On
RewriteRule ^/ /dummyprefix [PT]

WSGIScriptAlias /dummyprefix /path/to/script.wsgi

The real cause of the issue with the original RewriteRule is that the "." metacharacter doesn't really match all characters. Indeed, it doesn't match a newline. So, in order to match the full URL, including the evil encoded newline in the middle, one has to write an explicit range covering all possible characters:

AllowEncodedSlashes On

RewriteEngine On
RewriteRule ^/([\x00-\xff]*)$ /dummyprefix/$1 [PT]

WSGIScriptAlias /dummyprefix /path/to/script.wsgi

Whoops. Now a typical sysadmin probably won't understand the need (or will forget the reason) behind such a complex configuration for a seemingly simple issue of passing all URLs to a single script. And I am still not 100% sure that all valid URLs are really handled by the script. Maybe I should have started with a different web server, the one that doesn't have a polluted namespace in virtual hosts by default.

Update: In the comments, the following was suggested:

AllowEncodedSlashes On
WSGIHandlerScript wsgi-handler /path/to/handler.wsgi
SetHandler wsgi-handler

Here handler.wsgi would be a typical WSGI script, except that it provides the "handle_request" callable object instead of "application".

This solution looks elegant and simple, but, if PHP is also installed on the same server, it is wrong (thus proving the complexity of the problem and the fragility of Apache URL namespace). It misses URLs that end in .php (even though the document root s empty). So, back to the ugly solution based on mod_rewrite.

Linux audio sucks

because of bugs in drivers, libraries, and applications. Here are some of them. They are quite real, because they interfere with my not-so-unrealistic requirements (play music through headphones connected to the onboard HD audio chip, make some noise with the PC speaker when someone calls me via SIP, and copy sound from my TV tuner to the HD audio chip when I watch TV).

• In a driver. Take, for example, saa7134-alsa (a kernel module that allows recording audio from SAA713X-based TV tuners). It advertises support for the following sample rates: 32 kHz and 48 kHz. What it fails to mention is that it doesn't support recording at 48 kHz except for the case of "original SAA7134 with the MIXER_ADDR_LINE2 capture source". In all other cases (e.g., SAA7133 with the MIXER_ADDR_TVTUNER capture source), it gives out 32 kHz samples, but mislabels them as 48 kHz. Thus, applications such as MPlayer have to be configured to use only the 32 kHz sample rate. Not a big problem if such configuration is possible (i.e.: if not using PulseAudio together with HAL), but still a bug.
• In the ALSA library. Today I discovered that the PC speaker at home no longer plays ring tones in SIP clients. It did work before, but I can no longer find a working revision. While testing this sound device with speaker-test, I found this floating point exception inside the ALSA library.
• In applications. The most frequent bug is that an application doesn't have a way to use an arbitrary ALSA device, and instead has a drop-down box with pre-defined choices. Such applications often cannot work with Bluetooth headsets (that I don't have), FireWire audio cards (that I don't have either), and through non-default PulseAudio devices (and this has bit me when I tried to use Linphone with PulseAudio - I couldn't configure it to use the PC speaker through PulseAudio). What's even worse is that HAL endorses this faulty "drop-down box" enumeration scheme.