A case of network throughput optimization

The company that I work for has servers in several countries, including Germany, China, USA and Malaysia. We run MySQL with replication, and also sometimes need to copy images of virtual machines or LXC containers between servers. And, until recently, this was painfully slow, except between Germany and USA. We often resorted to recreating virtual machines and containers from the same template and doing the same manipulations, instead of just copying the result (e.g. using rsync or scp). We often received Munin alerts about MySQL replication not working well (i.e.: a test UPDATE that is done every two minutes on the master is not visible on the slave), and could not do anything about it. Because, well, it is just a very slow (stabilizes at 5 Mbit/s or so between USA and Malaysia, and even worse between China and anything else) network, and it is not our network.

So, it looked sad, except that raw UDP tests performed using iperf indicated much higher bandwidth (95 Mbit/s between USA and Malaysia, with only 0.034% packet loss) than what was available for scp or for MySQL replication between the same servers. So, it was clearly the case that the usual "don't tune anything" advice is questionable here, and system could, in theory, work better.

For the record, the latency, as reported by ping between the servers in USA and Malaysia, is 217 ms.

The available guides for Linux network stack tuning usually begin with sysctls regarding various buffer sizes. E.g., setting net.core.rmem_max and net.core.wmem_max to bigger values based on the bandwidth-delay product. In my case, the estimated bandwidth-delay product (which is the same as the amount of data in flight) would be about 2.7 megabytes. So, setting both to 8388608 and retesting with a larger TCP window size (4 M) should be logical. Except, it didn't really work. The throughput was only 8 Mbit/s instead of 5. I didn't try to modify net.ipv4.tcp_rmem or net.ipv4.tcp_wmem because the default values were already of the correct order of magnitude.

Other guides, including the official one from RedHat, talk about things like NIC ring buffers, interrupts, adapter queues and offloading. But these things are relevant for multi-gigabit networks, not for the mere 95 Mbit/s that we are aiming at.

The thing that actually helped was to change the TCP congestion control algorithm. This algorithm is what decides when to speed up data transmission and when to slow it down.

Linux comes with many modules that implement TCP congestion control algorithms. And, in newer kernels, there are new algorithms and some improvements in the old ones. So, it pays off to install a new kernel. For Ubuntu 16.04, this means installing the linux-generic-hwe-16.04-edge package.

The available modules are in /lib/modules/`uname -r`/kernel/net/ipv4/ directory. Here is how to load them all, for testing purposes:

cd /lib/modules/`uname -r`/kernel/net/ipv4/
for mod in tcp_*.ko ; do modprobe -v ${mod%.ko} ; done

For each of the loaded congestion control algorithms, it is possible to run iperf with the --linux-congestion parameter to benchmark it. Here are the results in my case, as reported by the server, with 4 M window (changed by the kernel to 8 M).

bbr: 56.7 Mbits/sec
bic: 24.5 Mbits/sec
cdg: 0.891 Mbits/sec
cubic: 8.38 Mbits/sec
dctcp: 17.6 Mbits/sec
highspeed: 1.50 Mbits/sec
htcp: 3.55 Mbits/sec
hybla: 20.6 Mbits/sec
illinois: 7.24 Mbits/sec
lp: 2.13 Mbits/sec
nv: 1.47 Mbits/sec
reno: 2.36 Mbits/sec
scalable: 2.50 Mbits/sec
vegas: 1.51 Mbits/sec
veno: 1.70 Mbits/sec
westwood: 3.83 Mbits/sec
yeah: 3.20 Mbits/sec

The condition that the speeds mentioned above are from the server-side reports (iperf server is the receiver of the data) is important. The client always reports higher throughput. This happens because the kernel buffers client's data and says "it has been finished" even though a lot of data sits in the buffer still waiting to be sent. The server sees the actual duration of the transfer and is thus in the position to provide an accurate report.

A good question is whether a large window or net.core.rmem_max and net.core.wmem_max is really needed. I don't think that benchmarking all algorithms again makes sense, because bbr is the clear winner. Actually, for cdg, which is the worst algorithm according to the above benchmark, leaving the window size and r/wmem_max at their default values resulted in a speed boost to 6.53 Mbits/sec. And here are the results for bbr:

Default window size, default r/wmem_max: 56.0 Mbits/sec
Default window size (85 or 128 KB), 8M r/wmem_max: 55.4 Mbits/sec
4M window, 8M r/wmem_max: 56.7 Mbits/sec (copied from the above)

I.e.: in this case, the only tuning needed was to switch the TCP congestion control algorithm to something modern. We did not achieve the maximum possible throughput, but even this is a 10x improvement.

Here is how to make the changes persistent:

echo tcp_bbr > /etc/modules-load.d/tcp.conf
echo net.ipv4.tcp_congestion_control=bbr > /etc/sysctl.d/91-tcp.conf

There are some important notes regarding the bbr congestion control algorithm:

1. It is only available starting with linux-4.9.
2. In kernels before 4.13, it only operated correctly when combined with the "fq" qdisc.
3. There are also important fixes, regarding recovery from the idle state of the connection, that happened in the 4.13 timeframe.

In other words, just use the latest kernel.

I will not repeat the mechanism due to which bbr is good on high-latency high-throughput slightly-lossy networks. Google's presentations do it better. Google uses it for youtube and other services, and it needs to be present on sender's side only. And it eliminated MySQL replication alerts for us. So maybe you should use it, too?

Root filesystem snapshots and kernel upgrades

On my laptop (which is running Arch), I decided to have periodic snapshots of the filesystem, in order to revert bad upgrades (especially those involving a large and unknown set of interdependent packages) easily. My toolset for this task is LVM2 and Snapper. Yes, I know that LVM2 is kind-of discouraged, and Snapper also supports btrfs, but most of the points below apply to btrfs, too.

Snapper, when used with LVM2, requires not just LVM2, but thinly-provisioned LVM2 volumes. Fortunately, Arch can have root filesystem on such volumes, so this is not a problem.

So, I have /boot on /dev/sda1, LVM on LUKS on /dev/sda2, root on a thinly-provisioned logical volume, and /home on another thinly-provisioned volume. And also swap on a non-thinly-provisioned volume. A separate /boot partition is needed because boot loaders generally don't understand thinly-provisioned LVM volumes, especially on encrypted disks. A separate volume for /home is needed because I don't want my new files in /home to be lost if I revert the system to its old snapshot. The same need to make a separate volume applies to other directories that contain data that should be preserved, but there are no such directories on my laptop. They can appear if I install e.g PostgreSQL.

And now there is a problem. Rollback to a snapshot works, but only if there were no kernel updates between the time when the snapshot was taken and when an attempt to revert was made. The root cause is that the kernel image is in /boot, and loadable modules for it are in /usr/lib/modules. The modules are reverted, but the boot loader still loads a new kernel, which now has no corresponding modules.

There are two solutions: either revert the kernel and its initramfs, too, when reverting the root file system, or make sure that modules are not reverted. I have not investigated how to make the first option possible, even though it would be a perfect solution. However, I have tried to make sure that modules are not reverted, and I am not satisfied with the result.

The idea was to move modules to /boot/modules, and make this location available somehow as /usr/lib/modules. Here "somehow" can mean either a symlink, or a bind mount. A symlink doesn't work, because the kernel upgrade in Arch will restore it back to a directory. A bind mount doesn't work, either. The issue is that, by putting modules on non-root filesystem, one creates a circular dependency between local filesystem mounting and udev (this would apply to a symlink, too).

Indeed, systemd-udevd, on startup, maps the /usr/lib/modules/`uname -r`/modules.alias.bin file into memory. So, now it has a (real) dependency on /usr/lib/modules being mounted. However, mounting local filesystems from /etc/fstab sometimes depends on systemd-udevd, because of device nodes. So, bind-mounting /usr/lib/modules merely from /etc/fstab, using built-in systemd tools, cannot work.

But it can work from a wrapper that starts before the real init:

#!/bin/sh
mount -n /boot              # /dev/sda1 is in devtmpfs and doesn't need udev
mount -n /usr/lib/modules   # there is still a line in fstab about that
exec /sbin/init "$@" 

But that's ugly. In the end, I removed the wrapper, installed an old known-working "linux" package, made a copy of the kernel, its initramfs and modules, upgraded the kernel again, and put the saved files back, so that they are now not controlled by the package manager. So now I have a known good kernel down in the boot menu, and knowledge that its modules will always be present in my root filesystem if I don't revert further than up to today's state.

And now one final remark. Remember that I said: "The same need to make a separate volume applies to other directories that contain data that should be preserved"? There is a temptation to apply this to the whole /var directory, but that would be wrong. If a system is being reverted to its old snapshot, a package database (which is in /var/lib/pacman) should be reverted, too. But /var/lib/pacman is under /var.

The conclusion is that Linux plumbers should think a bit about this "revert the whole system" use case, and maybe move some directories.

Autoconf and defaults

Autoconf is based on a set of macros. these macros are put by developers into the configure.ac file, and the "autoconf" program generates a proper configure script from that. The configure script commonly accepts options such as --enable-foo and --with-foo. It is important that they behave in a sensible way.

The --with-foo argument is processed with the AC_ARG_WITH macro. It is documented as follows:

AC_ARG_WITH (package, help-string, [action-if-given], [action-if-not-given])

If the user gave configure the option --with-package or --without-package, run shell commands action-if-given. If neither option was given, run shell commands action-if-not-given. The name package indicates another software package that this program should work with. It should consist only of alphanumeric characters, dashes, plus signs, and dots.

The option's argument is available to the shell commands action-if-given in the shell variable withval, which is actually just the value of the shell variable named with_package, with any non-alphanumeric characters in package changed into ‘_’. You may use that variable instead, if you wish.

 Let's see what's wrong with this piece of configure.ac, found in lxdm-0.4.1:

AC_ARG_WITH(pam,AC_HELP_STRING([--without-pam],[build without pam]),
[],[AC_CHECK_LIB([pam], [pam_open_session])])

This does nothing if the --without-pam option is not given, and checks for the PAM library if the option is given. Looks right? No!

This also does nothing if the --with-pam option is given, thus, resulting in a build without PAM support!

Note that the documentation for AC_ARG_WITH even provides examples how to use the macro properly. Use them as a reference, and have the following checklist:

• Use the $withval or $with_package variable.
• Test it for values such as "yes", "no", "auto" and explicit path.
• Check the presence of the package unless it is explicitly disabled or the support is experimental.
• Fail if the external package is requested but not available.

Yes, this becomes verbose, but it is necessary for correctness.

The first point also applies to the AC_ARG_ENABLE macro:

AC_ARG_ENABLE(debug,         AS_HELP_STRING([--enable-debug],
                                  [Enable debugging (default: disabled)]),
                                [DEBUGGING=$enableval], [DEBUGGING=no])

I wrote a DTS encoder

Let me announce a piece of software that I have published a week ago: dcaenc, an open-source DTS encoder. The package contains sources for a shared library, a command-line tool and an ALSA plugin.

DTS is one of the compressed formats that allow transfer of multichannel (e.g., 5.1) audio over SPDIF connections. The other common format is AC3. The SPDIF standard does not define a method for passing more than two channels of uncompressed PCM audio, so compression has to be used. Both AC3 and DTS are also used in DVD sound tracks.

Open-source decoders for both AC3 and DTS already exist: liba52 and libdca (side note: please don't use libdca, it is a security risk, there are some files that crash it or are decoded improperly). FFmpeg can also decode these formats. However, useful open-source encoders existed only for AC3: one in FFmpeg, and the other one (aften) based on it. The DTS "encoder" in FFmpeg was ported by someone else from my old proof-of-concept code that served as a tool to understand the DTS subband transform. It could only encode stereo PCM files into a valid DTS bitstream of the same bitrate, which is useless for any practical purpose. Now dcaenc provides a useful encoder that accepts multichannel sound and encodes it to the bitrate specified by the command line parameter.

As already mentioned, there are the following use cases for my encoder:

• On-the-fly encoding of multichannel PCM audio produced by arbitrary ALSA applications (e.g. games) for transmission via SPDIF
• Creation of DVD soundtracks and DTS CDs.

Some people ask me why I didn't integrate my encoder into FFmpeg instead of releasing it as a standalone package. Indeed, there are faster implementations of the basic DSP building blocks in FFmpeg, and the criticism that I reinvented a lot of wheels is valid. Integration with FFmpeg is indeed a desired long-term goal.

There are still several reasons why I decided not to integrate right from the beginning. First, I don't think that my work is in the necessary shape for integration yet. E.g., in FFmpeg, floating-point codecs are preferred, while my library currently uses fixed-point (I thought it would be beneficial for porting to the ARM architecture). Second and the most important reason: when the encoder is standalone, users can get it immediately and use it, without the hassle of replacing the distribution-provided FFmpeg package and potentially breaking distribution-provided software such as VLC that depends on it. Third, if I know that I wrote all the code myself, I can sell LGPL exceptions.

While dcaenc already produces "transparent" output at 1411 or 1536 kilobits per second, there is still room for quality improvement at lower bitrates. This is because the library does not yet use all compression possibilities offered by the DTS standard. I am going to implement at least linear prediction (incorrectly called ADPCM in the specification) and channel coupling in the future versions. Stay tuned!

I am going to Desktop Summit 2011

I am going to Desktop Summit. Actually, I have already arrived in Berlin.

At the summit, I will mostly follow the Platform track (Rm2002) and then try to participate in BoFs.

Package dependencies

Let's start with this fact: linphone uses the ffmpeg library to encode video. How can various package managers be informed about this fact by maintainers while building the linphone package?

In Arch Linux, the dependency information is expressed in the PKGBUILD file in a very straightforward way:

depends=('alsa-lib' 'ffmpeg>=20080715' 'gtk2' 'libexosip2>=3.1.0' 'speex>=1.1.12')

And this propagates in the same form into the binary package of linphone. But, is it really what is needed to prevent installation of a broken package? No.

The linphone binary from the package uses the libavcodec.so.51 library, not just "any ffmpeg >=20080715". So the linphone package will break when the ffmpeg package starts providing libavcodec.so.52 (and this already happened in the testing repository). Let's see how other package managers deal with the situation.

RPM-based distributions have to add this line to the linphone.spec script:

BuildRequires:  ffmpeg-dev

Then, RPM will build the linphone package and examine the resulting binaries. It will notice that the linphone binary uses the libavcodec.so.51 library and will add the dependency on that library (not on some abstract "ffmpeg" package). This way, if the new ffmpeg package no longer includes libavcodec.so.51, RPM will notice that the linphone package now has an unsatisfied dependency, print an error message and refuse to upgrade ffmpeg.

This is better than the Arch Linux way, but requires some external tool to know that libavcodec.so.51 is actually in the ffmpeg package. Otherwise, how would the "install linphone with all dependencies" request be handled?

Now let's talk about Debian and its derivatives. They also have a notion of a build-time dependency, and it is expressed in the debian/control file:

Build-Depends: ..., libavcodec-dev, ...

When the package is built, debhelper also notices that the resulting linphone binary depends on libavcodec.so.51, and searches the installed packages for this library. As a result, the dependency on the found package is added automatically to the binary package of linphone. This looks similar to the Arch Linux case, but Debian has a policy that requires the shared library version to be mentioned in the package name. I.e., linphone gets a dependency on "libavcodec51" package, while "libavcodec52" is a completely different package that (assuming no bugs) can be installed in parallel with libavcodec51, thus allowing the rebuilds of other packages for the ffmpeg upgrade to take place gradually, without any hurry and without the broken intermediate state.