Namespace issues in Apache

Some time ago I was given a task to write a Python WSGI script that handles all (really all!) URLs on a given Apache-based virtual host. Doesn't that sound easy? Here is the "obvious" part of the Apache configuration file for that virtual host:

WSGIScriptAlias / /path/to/script.wsgi

Except that it is slightly wrong. The problem is that the script really should handle all URLS, even "evil" ones, without any omissions. In the above form, it doesn't.

First, if the URL contains a percent-encoded slash (e.g., as in http://example.org/foo%2fbar), Apache gives a 404 error by default, without even calling the script. Solution:

AllowEncodedSlashes On

Wait, there is more! Regular aliases have higher priority than our WSGIScriptAlias, and that there are some default aliases usually set by distributions. Due to that, http://example.org/icons/folder.gif will map to a static file, not to the WSGI script. There is a wishlist bug in Apache that one cannot easily remove aliases from the namespace.

The solution (or, more precisely, a bad hack) that I found is to use mod_rewrite. By adding some prefix to all URLs (and, of course, dealing with it in the script), one makes sure that the existing aliases are not hit:

AllowEncodedSlashes On

RewriteEngine On
RewriteRule ^/(.*)$ /dummyprefix/$1 [PT]

WSGIScriptAlias /dummyprefix /path/to/script.wsgi

Still, this doesn't work (gives a 404 error instead of calling the script) on URLs that include some bad characters such as newlines, e.g. http://example.org/foo%0abar . Removing the dollar sign from the pattern fixes the 404 error, calls the script, but it then receives a truncated PATH_INFO in the environment. Still, SCRIPT_URI is correct, so this may be enough. In fact, if the script doesn't care about PATH_INFO, even this works:

AllowEncodedSlashes On

RewriteEngine On
RewriteRule ^/ /dummyprefix [PT]

WSGIScriptAlias /dummyprefix /path/to/script.wsgi

The real cause of the issue with the original RewriteRule is that the "." metacharacter doesn't really match all characters. Indeed, it doesn't match a newline. So, in order to match the full URL, including the evil encoded newline in the middle, one has to write an explicit range covering all possible characters:

AllowEncodedSlashes On

RewriteEngine On
RewriteRule ^/([\x00-\xff]*)$ /dummyprefix/$1 [PT]

WSGIScriptAlias /dummyprefix /path/to/script.wsgi

Whoops. Now a typical sysadmin probably won't understand the need (or will forget the reason) behind such a complex configuration for a seemingly simple issue of passing all URLs to a single script. And I am still not 100% sure that all valid URLs are really handled by the script. Maybe I should have started with a different web server, the one that doesn't have a polluted namespace in virtual hosts by default.

Update: In the comments, the following was suggested:

AllowEncodedSlashes On
WSGIHandlerScript wsgi-handler /path/to/handler.wsgi
SetHandler wsgi-handler

Here handler.wsgi would be a typical WSGI script, except that it provides the "handle_request" callable object instead of "application".

This solution looks elegant and simple, but, if PHP is also installed on the same server, it is wrong (thus proving the complexity of the problem and the fragility of Apache URL namespace). It misses URLs that end in .php (even though the document root s empty). So, back to the ugly solution based on mod_rewrite.

Autoconf and defaults

Autoconf is based on a set of macros. these macros are put by developers into the configure.ac file, and the "autoconf" program generates a proper configure script from that. The configure script commonly accepts options such as --enable-foo and --with-foo. It is important that they behave in a sensible way.

The --with-foo argument is processed with the AC_ARG_WITH macro. It is documented as follows:

AC_ARG_WITH (package, help-string, [action-if-given], [action-if-not-given])

If the user gave configure the option --with-package or --without-package, run shell commands action-if-given. If neither option was given, run shell commands action-if-not-given. The name package indicates another software package that this program should work with. It should consist only of alphanumeric characters, dashes, plus signs, and dots.

The option's argument is available to the shell commands action-if-given in the shell variable withval, which is actually just the value of the shell variable named with_package, with any non-alphanumeric characters in package changed into ‘_’. You may use that variable instead, if you wish.

 Let's see what's wrong with this piece of configure.ac, found in lxdm-0.4.1:

AC_ARG_WITH(pam,AC_HELP_STRING([--without-pam],[build without pam]),
[],[AC_CHECK_LIB([pam], [pam_open_session])])

This does nothing if the --without-pam option is not given, and checks for the PAM library if the option is given. Looks right? No!

This also does nothing if the --with-pam option is given, thus, resulting in a build without PAM support!

Note that the documentation for AC_ARG_WITH even provides examples how to use the macro properly. Use them as a reference, and have the following checklist:

• Use the $withval or $with_package variable.
• Test it for values such as "yes", "no", "auto" and explicit path.
• Check the presence of the package unless it is explicitly disabled or the support is experimental.
• Fail if the external package is requested but not available.

Yes, this becomes verbose, but it is necessary for correctness.

The first point also applies to the AC_ARG_ENABLE macro:

AC_ARG_ENABLE(debug,         AS_HELP_STRING([--enable-debug],
                                  [Enable debugging (default: disabled)]),
                                [DEBUGGING=$enableval], [DEBUGGING=no])

I wrote a DTS encoder

Let me announce a piece of software that I have published a week ago: dcaenc, an open-source DTS encoder. The package contains sources for a shared library, a command-line tool and an ALSA plugin.

DTS is one of the compressed formats that allow transfer of multichannel (e.g., 5.1) audio over SPDIF connections. The other common format is AC3. The SPDIF standard does not define a method for passing more than two channels of uncompressed PCM audio, so compression has to be used. Both AC3 and DTS are also used in DVD sound tracks.

Open-source decoders for both AC3 and DTS already exist: liba52 and libdca (side note: please don't use libdca, it is a security risk, there are some files that crash it or are decoded improperly). FFmpeg can also decode these formats. However, useful open-source encoders existed only for AC3: one in FFmpeg, and the other one (aften) based on it. The DTS "encoder" in FFmpeg was ported by someone else from my old proof-of-concept code that served as a tool to understand the DTS subband transform. It could only encode stereo PCM files into a valid DTS bitstream of the same bitrate, which is useless for any practical purpose. Now dcaenc provides a useful encoder that accepts multichannel sound and encodes it to the bitrate specified by the command line parameter.

As already mentioned, there are the following use cases for my encoder:

• On-the-fly encoding of multichannel PCM audio produced by arbitrary ALSA applications (e.g. games) for transmission via SPDIF
• Creation of DVD soundtracks and DTS CDs.

Some people ask me why I didn't integrate my encoder into FFmpeg instead of releasing it as a standalone package. Indeed, there are faster implementations of the basic DSP building blocks in FFmpeg, and the criticism that I reinvented a lot of wheels is valid. Integration with FFmpeg is indeed a desired long-term goal.

There are still several reasons why I decided not to integrate right from the beginning. First, I don't think that my work is in the necessary shape for integration yet. E.g., in FFmpeg, floating-point codecs are preferred, while my library currently uses fixed-point (I thought it would be beneficial for porting to the ARM architecture). Second and the most important reason: when the encoder is standalone, users can get it immediately and use it, without the hassle of replacing the distribution-provided FFmpeg package and potentially breaking distribution-provided software such as VLC that depends on it. Third, if I know that I wrote all the code myself, I can sell LGPL exceptions.

While dcaenc already produces "transparent" output at 1411 or 1536 kilobits per second, there is still room for quality improvement at lower bitrates. This is because the library does not yet use all compression possibilities offered by the DTS standard. I am going to implement at least linear prediction (incorrectly called ADPCM in the specification) and channel coupling in the future versions. Stay tuned!

I am going to Desktop Summit 2011

I am going to Desktop Summit. Actually, I have already arrived in Berlin.

At the summit, I will mostly follow the Platform track (Rm2002) and then try to participate in BoFs.

The case of a non-raised exception

As you probably know, python uses exceptions for error handling. It is considered a good style to avoid adding error-handling code in the form of conditional statements. Instead, one should rely on the fact that an appropriate exception is raised once an error condition is detected, and caught when it can be dealt with.

So, let'a assume that you are given a task of downloading a file given the URL and the file name on disk, using Python. You may want to write the following code and hope that you don't have to add any error-handling because (as you think) all errors that can happen are either network errors or file write errors, and those two types of errors already raise exceptions for you.

#!/usr/bin/python

import urllib2
import sys
import socket

def download(url, fname):
    net = urllib2.urlopen(url)
    f = open(fname, "wb")
    
    while True:
        data = net.read(4096)
        if not data:
            break
        f.write(data)
    
    net.close()
    f.close()

if __name__ == "__main__":
    if len(sys.argv) != 3:
        print "Usage: download.py URL filename"
    
    url = sys.argv[1]
    fname = sys.argv[2]
    
    socket.setdefaulttimeout(30)
    
    download(url, fname)

Indeed, this code downloads existing files via HTTP just fine. Also, it provides sensible tracebacks for non-existing hosts, 404 errors, full-disk situations, and socket timeouts. So, it looks like the result of calling the download() fnction is either a successfully downloaded file, or an exception that the other part of the application will likely be able to deal with.

But actually, it only looks like this. Consider a situation when the HTTP server closes the connection gracefully at the TCP level, but prematurely. You can test this by starting your own Apache web server, putting a large file there, and calling "apache2ctl restart" while the client is downloading the file. Result: an incompletely downloaded file, and no exceptions.

I don't know if it should be considered a bug in urllib2 or in the example download() function above. In fact, urllib2 could have noticed the mismatch of the total number of bytes before the EOF and the value in the Content-Length HTTP header.

Here is a version of the download() function that detects incomplete downloads based on the Content-Length header:

def download(url, fname):
    net = urllib2.urlopen(url)
    contentlen = net.info().get("Content-Length", "")
    f = open(fname, "wb")
    datalen = 0
    
    while True:
        data = net.read(4096)
        if not data:
            break
        f.write(data)
 datalen += len(data)
    
    net.close()
    f.close()

    try:
        contentlen = int(contentlen)
    except ValueError:
        contentlen = None

    if contentlen is not None and contentlen != datalen:
        raise urllib2.URLError("Incomplete download")

Writing systemd service files

Today I tried to convert a Gentoo initscript for VDE to a systemd service file. This post documents the required steps. VDE has been chosen as an example because it is a simple daemon (commonly used for communication between several instances of QEMU and the host) that illustrates the matter well.

Background

For those who don't know, systemd is a next-generation replacement for /sbin/init written by Lennart Poettering. While systemd supports traditional initscripts, service files are its native form of configuration. Service files have a structure similar to that of desktop files or Windows ini files. Their syntax is well documented in the manual pages.

Initscripts typically contain boilerplate code that checks whether the daemon is already running or figures out which process to kill when the daemon has to be stopped. Tools like start-stop-daemon help, but systemd reduces the syntax overhead to the minimum. This simplification occurs because service files specify what should be done, not how it should be done. I.e., unlike initscripts, they follow the declarative style and are not programs.

Simple services

Here is the simplest possible service file that starts VDE with options that work for me on my computer. Save it as /etc/systemd/system/vde.service:

[Unit]
Description=Virtual Distributed Ethernet

[Service]
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu

[Install]
WantedBy=multi-user.target

Note the difference from a traditional init script: for simplicity, vde_switch is started in such a way that it doesn't become a daemon. In fact, it is possible to start real daemons that fork, you just have to tell systemd about that:

[Unit]
Description=Virtual Distributed Ethernet

[Service]
Type=forking
# The PID file is optional, but recommended in the manpage
# "so that systemd can identify the main process of the daemon"
PIDFile=/var/run/vde.pid
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu \
 --daemon --pidfile /var/run/vde.pid

[Install]
WantedBy=multi-user.target

The difference is in the way the dependencies are handled. If some other services depend on vde.service, then, in the first example, systemd will be able to run them as soon as it starts vde_switch. In the second example, systemd will wait until vde_switch forks. The difference matters, because vde_switch creates its control socket after starting, but before forking. So, in the first example, there is some chance that systemd will start something that tries to connect to the socket before vde_switch creates it.

Automatic restarts

Let's also add the proper dependency on the system logger and tell systemd to restart vde_switch if it crashes due to an uncaught signal (although it never happened to me):

[Unit]
Description=Virtual Distributed Ethernet
After=syslog.target

[Service]
Type=forking
PIDFile=/var/run/vde.pid
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu \
 --daemon --pidfile /var/run/vde.pid
Restart=on-abort

[Install]
WantedBy=multi-user.target

Now try to start and crash the daemon:

home ~ # systemctl start vde.service
home ~ # systemctl status vde.service
vde.service - Virtual Distributed Ethernet
   Loaded: loaded (/etc/systemd/system/vde.service)
   Active: active (running) since Tue, 04 Jan 2011 22:08:10 +0500;
15s ago
  Process: 31434 (/usr/bin/vde_switch --tap tap0...,
code=exited, status=0/SUCCESS)
 Main PID: 31435 (vde_switch)
   CGroup: name=systemd:/system/vde.service
    └ 31435 /usr/bin/vde_switch --tap tap0...
home ~ # kill -SEGV 31435
home ~ # systemctl status vde.service
vde.service - Virtual Distributed Ethernet
   Loaded: loaded (/etc/systemd/system/vde.service)
   Active: failed since Tue, 04 Jan 2011 22:11:27 +0500;
4s ago
  Process: 31503 (/usr/bin/vde_switch --tap tap0...,
code=exited, status=0/SUCCESS)
 Main PID: 31504 (code=exited, status=1/FAILURE)
   CGroup: name=systemd:/system/vde.service

I.e., restarting didn't work. The system log tells us why:

Jan  4 22:11:27 home vde_switch[31504]: Error in pidfile
creation: File exists

So, VDE has a bug in its pidfile creation. There are two ways how one can deal with this: either tell systemd to remove the PID file before starting vde_switch, or drop the PID file altogether (because vde_switch has exactly one process, there can be no confusion which process is the main one). Both ways work. Here is how to implement the first alternative:

[Unit]
Description=Virtual Distributed Ethernet
After=syslog.target

[Service]
Type=forking
PIDFile=/var/run/vde.pid
# Note the -f: don't fail if there is no PID file
ExecStartPre=/bin/rm -f /var/run/vde.pid
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu \
 --daemon --pidfile /var/run/vde.pid
Restart=on-abort

[Install]
WantedBy=multi-user.target

And here is the second alternative:

[Unit]
Description=Virtual Distributed Ethernet
After=syslog.target

[Service]
Type=forking
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu \
 --daemon
Restart=on-abort

[Install]
WantedBy=multi-user.target

Configuration

Many initscripts come with configuration files that allow the user to customize how the daemon is started. For example, some users may want to pass the --hub argument to vde_switch, and others may want to enable the management socket. So, in Gentoo, the traditional configuration file for the initscript looks like this:

# load the tun module
VDE_MODPROBE_TUN="yes"
# virtual tap networking device to be used for vde
VDE_TAP="tap0"
# mode and group for the socket
VDE_SOCK_CHMOD="770"
VDE_SOCK_CHOWN=":qemu"

# This is the actual options string passed to VDE.
# Change this at your own risk.
VDE_OPTS="--tap ${VDE_TAP} -daemon"

Systemd service files typically use the EnvironmentFile key to provide users with a file where they can put their preferences regarding the service.

Traditional initscripts source their configuration files. Thus, any syntax construction supported by /bin/sh will work in the configuration file. In the example above, we see comments, assignment of values to variables, and reusing the values in later assignments. Systemd uses a different syntax from bash, so please resist the temptation to reuse the same configuration file for the traditional initscript and the service file. Resist even though some service files in the Gentoo systemd overlay do use the same configuration files as the corresponding traditional initscripts -- they are just buggy. Let me explain this in more detail.

Of course, the configuration file example above is not suitable for systemd because variable interpolation is not supported in systemd environment files. But let's suppose that we want to invent something that is suitable both as a bash script fragment and a systemd environment file, and still allows the user to configure vde_switch according to his wishes.

Let's focus on the VDE_OPTS variable only, as it is the only thing that matters. Indeed, module loading can be done directly by systemd (man modules-load.d), and the options related to the socket group and octal permissions can be expressed using vde_switch command line options, as illustrated in the examples above. Since the value of the VDE_OPTS variable can contain spaces, we have to quote it if we want bash to be able to understand what we mean:

VDE_OPTS="--tap tap0 --mode 0660 --dirmode 0750 --group qemu"

Without quotes, bash would interpret this as follows: with the variable VDE_OPTS that has value "--tap" in the environment, start the "tap0" process and pass 6 parameters to it. So, quotes are essential here.

Let's try to use this variable from the service file. We want it to be split by the spaces, so that each part becomes a separate vde_switch parameter. So, according to the systemd.service manpage, we have to use the $VDE_OPTS form, not ${VDE_OPTS}. So here is what we have:

[Unit]
Description=Virtual Distributed Ethernet
After=syslog.target

[Service]
Type=forking
EnvironmentFile=/etc/conf.d/vde2
ExecStart=/usr/bin/vde_switch --daemon $VDE_OPTS
Restart=on-abort

[Install]
WantedBy=multi-user.target

Result: no "tap0" interface and wrong permissions on the control socket. This happened because quotes play a special role in systemd environment files, different from their role in bash scripts. For systemd, they mean that the spaces inside them should not be treated as argument separators. So, all the arguments in $VDE_OPTS were passed to vde_switch as one long argument. No surprise that it didn't work.

In fact, the service file is correct. Its configuration just can't be made compatible with bash, because it is in a different language. The service works with the following configuration file (alas, incompatible with bash):

VDE_OPTS=--tap tap0 --mode 0660 --dirmode 0750 --group qemu

Conclusion

Let's hope that all of the above will get you started writing your own systemd service files. Since there are many initscripts in Gentoo still not converted, the project needs your help.

Thanks

The following people from #systemd IRC channel on freenode provided me with valuable support: MK_FG, zdzichuBG, miti1.