I am going to Desktop Summit. Actually, I have already arrived in Berlin.
At the summit, I will mostly follow the Platform track (Rm2002) and then try to participate in BoFs.
Friday, August 5, 2011
I am going to Desktop Summit 2011
I am going to Desktop Summit. Actually, I have already arrived in Berlin.
At the summit, I will mostly follow the Platform track (Rm2002) and then try to participate in BoFs.
Saturday, March 12, 2011
The case of a non-raised exception
As you probably know, python uses exceptions for error handling. It is considered a good style to avoid adding error-handling code in the form of conditional statements. Instead, one should rely on the fact that an appropriate exception is raised once an error condition is detected, and caught when it can be dealt with.
So, let'a assume that you are given a task of downloading a file given the URL and the file name on disk, using Python. You may want to write the following code and hope that you don't have to add any error-handling because (as you think) all errors that can happen are either network errors or file write errors, and those two types of errors already raise exceptions for you.
Indeed, this code downloads existing files via HTTP just fine. Also, it provides sensible tracebacks for non-existing hosts, 404 errors, full-disk situations, and socket timeouts. So, it looks like the result of calling the download() fnction is either a successfully downloaded file, or an exception that the other part of the application will likely be able to deal with.
But actually, it only looks like this. Consider a situation when the HTTP server closes the connection gracefully at the TCP level, but prematurely. You can test this by starting your own Apache web server, putting a large file there, and calling "apache2ctl restart" while the client is downloading the file. Result: an incompletely downloaded file, and no exceptions.
I don't know if it should be considered a bug in urllib2 or in the example download() function above. In fact, urllib2 could have noticed the mismatch of the total number of bytes before the EOF and the value in the Content-Length HTTP header.
Here is a version of the download() function that detects incomplete downloads based on the Content-Length header:
So, let'a assume that you are given a task of downloading a file given the URL and the file name on disk, using Python. You may want to write the following code and hope that you don't have to add any error-handling because (as you think) all errors that can happen are either network errors or file write errors, and those two types of errors already raise exceptions for you.
#!/usr/bin/python
import urllib2
import sys
import socket
def download(url, fname):
net = urllib2.urlopen(url)
f = open(fname, "wb")
while True:
data = net.read(4096)
if not data:
break
f.write(data)
net.close()
f.close()
if __name__ == "__main__":
if len(sys.argv) != 3:
print "Usage: download.py URL filename"
url = sys.argv[1]
fname = sys.argv[2]
socket.setdefaulttimeout(30)
download(url, fname)
Indeed, this code downloads existing files via HTTP just fine. Also, it provides sensible tracebacks for non-existing hosts, 404 errors, full-disk situations, and socket timeouts. So, it looks like the result of calling the download() fnction is either a successfully downloaded file, or an exception that the other part of the application will likely be able to deal with.
But actually, it only looks like this. Consider a situation when the HTTP server closes the connection gracefully at the TCP level, but prematurely. You can test this by starting your own Apache web server, putting a large file there, and calling "apache2ctl restart" while the client is downloading the file. Result: an incompletely downloaded file, and no exceptions.
I don't know if it should be considered a bug in urllib2 or in the example download() function above. In fact, urllib2 could have noticed the mismatch of the total number of bytes before the EOF and the value in the Content-Length HTTP header.
Here is a version of the download() function that detects incomplete downloads based on the Content-Length header:
def download(url, fname):
net = urllib2.urlopen(url)
contentlen = net.info().get("Content-Length", "")
f = open(fname, "wb")
datalen = 0
while True:
data = net.read(4096)
if not data:
break
f.write(data)
datalen += len(data)
net.close()
f.close()
try:
contentlen = int(contentlen)
except ValueError:
contentlen = None
if contentlen is not None and contentlen != datalen:
raise urllib2.URLError("Incomplete download")
Tuesday, January 4, 2011
Writing systemd service files
Today I tried to convert a Gentoo initscript for VDE to a systemd service file. This post documents the required steps. VDE has been chosen as an example because it is a simple daemon (commonly used for communication between several instances of QEMU and the host) that illustrates the matter well.
For those who don't know, systemd is a next-generation replacement for /sbin/init written by Lennart Poettering. While systemd supports traditional initscripts, service files are its native form of configuration. Service files have a structure similar to that of desktop files or Windows ini files. Their syntax is well documented in the manual pages.
Initscripts typically contain boilerplate code that checks whether the daemon is already running or figures out which process to kill when the daemon has to be stopped. Tools like start-stop-daemon help, but systemd reduces the syntax overhead to the minimum. This simplification occurs because service files specify what should be done, not how it should be done. I.e., unlike initscripts, they follow the declarative style and are not programs.
Here is the simplest possible service file that starts VDE with options that work for me on my computer. Save it as /etc/systemd/system/vde.service:
Note the difference from a traditional init script: for simplicity, vde_switch is started in such a way that it doesn't become a daemon. In fact, it is possible to start real daemons that fork, you just have to tell systemd about that:
The difference is in the way the dependencies are handled. If some other services depend on vde.service, then, in the first example, systemd will be able to run them as soon as it starts vde_switch. In the second example, systemd will wait until vde_switch forks. The difference matters, because vde_switch creates its control socket after starting, but before forking. So, in the first example, there is some chance that systemd will start something that tries to connect to the socket before vde_switch creates it.
Let's also add the proper dependency on the system logger and tell systemd to restart vde_switch if it crashes due to an uncaught signal (although it never happened to me):
Now try to start and crash the daemon:
I.e., restarting didn't work. The system log tells us why:
So, VDE has a bug in its pidfile creation. There are two ways how one can deal with this: either tell systemd to remove the PID file before starting vde_switch, or drop the PID file altogether (because vde_switch has exactly one process, there can be no confusion which process is the main one). Both ways work. Here is how to implement the first alternative:
And here is the second alternative:
Many initscripts come with configuration files that allow the user to customize how the daemon is started. For example, some users may want to pass the --hub argument to vde_switch, and others may want to enable the management socket. So, in Gentoo, the traditional configuration file for the initscript looks like this:
Systemd service files typically use the EnvironmentFile key to provide users with a file where they can put their preferences regarding the service.
Traditional initscripts source their configuration files. Thus, any syntax construction supported by /bin/sh will work in the configuration file. In the example above, we see comments, assignment of values to variables, and reusing the values in later assignments. Systemd uses a different syntax from bash, so please resist the temptation to reuse the same configuration file for the traditional initscript and the service file. Resist even though some service files in the Gentoo systemd overlay do use the same configuration files as the corresponding traditional initscripts -- they are just buggy. Let me explain this in more detail.
Of course, the configuration file example above is not suitable for systemd because variable interpolation is not supported in systemd environment files. But let's suppose that we want to invent something that is suitable both as a bash script fragment and a systemd environment file, and still allows the user to configure vde_switch according to his wishes.
Let's focus on the VDE_OPTS variable only, as it is the only thing that matters. Indeed, module loading can be done directly by systemd (man modules-load.d), and the options related to the socket group and octal permissions can be expressed using vde_switch command line options, as illustrated in the examples above. Since the value of the VDE_OPTS variable can contain spaces, we have to quote it if we want bash to be able to understand what we mean:
Without quotes, bash would interpret this as follows: with the variable VDE_OPTS that has value "--tap" in the environment, start the "tap0" process and pass 6 parameters to it. So, quotes are essential here.
Let's try to use this variable from the service file. We want it to be split by the spaces, so that each part becomes a separate vde_switch parameter. So, according to the systemd.service manpage, we have to use the $VDE_OPTS form, not ${VDE_OPTS}. So here is what we have:
Result: no "tap0" interface and wrong permissions on the control socket. This happened because quotes play a special role in systemd environment files, different from their role in bash scripts. For systemd, they mean that the spaces inside them should not be treated as argument separators. So, all the arguments in $VDE_OPTS were passed to vde_switch as one long argument. No surprise that it didn't work.
In fact, the service file is correct. Its configuration just can't be made compatible with bash, because it is in a different language. The service works with the following configuration file (alas, incompatible with bash):
Let's hope that all of the above will get you started writing your own systemd service files. Since there are many initscripts in Gentoo still not converted, the project needs your help.
The following people from #systemd IRC channel on freenode provided me with valuable support: MK_FG, zdzichuBG, miti1.
Background
For those who don't know, systemd is a next-generation replacement for /sbin/init written by Lennart Poettering. While systemd supports traditional initscripts, service files are its native form of configuration. Service files have a structure similar to that of desktop files or Windows ini files. Their syntax is well documented in the manual pages.
Initscripts typically contain boilerplate code that checks whether the daemon is already running or figures out which process to kill when the daemon has to be stopped. Tools like start-stop-daemon help, but systemd reduces the syntax overhead to the minimum. This simplification occurs because service files specify what should be done, not how it should be done. I.e., unlike initscripts, they follow the declarative style and are not programs.
Simple services
Here is the simplest possible service file that starts VDE with options that work for me on my computer. Save it as /etc/systemd/system/vde.service:
[Unit] Description=Virtual Distributed Ethernet [Service] ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \ --dirmode 0750 --group qemu [Install] WantedBy=multi-user.target
Note the difference from a traditional init script: for simplicity, vde_switch is started in such a way that it doesn't become a daemon. In fact, it is possible to start real daemons that fork, you just have to tell systemd about that:
[Unit] Description=Virtual Distributed Ethernet [Service] Type=forking # The PID file is optional, but recommended in the manpage # "so that systemd can identify the main process of the daemon" PIDFile=/var/run/vde.pid ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \ --dirmode 0750 --group qemu \ --daemon --pidfile /var/run/vde.pid [Install] WantedBy=multi-user.target
The difference is in the way the dependencies are handled. If some other services depend on vde.service, then, in the first example, systemd will be able to run them as soon as it starts vde_switch. In the second example, systemd will wait until vde_switch forks. The difference matters, because vde_switch creates its control socket after starting, but before forking. So, in the first example, there is some chance that systemd will start something that tries to connect to the socket before vde_switch creates it.
Automatic restarts
Let's also add the proper dependency on the system logger and tell systemd to restart vde_switch if it crashes due to an uncaught signal (although it never happened to me):
[Unit] Description=Virtual Distributed Ethernet After=syslog.target [Service] Type=forking PIDFile=/var/run/vde.pid ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \ --dirmode 0750 --group qemu \ --daemon --pidfile /var/run/vde.pid Restart=on-abort [Install] WantedBy=multi-user.target
Now try to start and crash the daemon:
home ~ # systemctl start vde.service
home ~ # systemctl status vde.service
vde.service - Virtual Distributed Ethernet
Loaded: loaded (/etc/systemd/system/vde.service)
Active: active (running) since Tue, 04 Jan 2011 22:08:10 +0500;
15s ago
Process: 31434 (/usr/bin/vde_switch --tap tap0...,
code=exited, status=0/SUCCESS)
Main PID: 31435 (vde_switch)
CGroup: name=systemd:/system/vde.service
└ 31435 /usr/bin/vde_switch --tap tap0...
home ~ # kill -SEGV 31435
home ~ # systemctl status vde.service
vde.service - Virtual Distributed Ethernet
Loaded: loaded (/etc/systemd/system/vde.service)
Active: failed since Tue, 04 Jan 2011 22:11:27 +0500;
4s ago
Process: 31503 (/usr/bin/vde_switch --tap tap0...,
code=exited, status=0/SUCCESS)
Main PID: 31504 (code=exited, status=1/FAILURE)
CGroup: name=systemd:/system/vde.service
I.e., restarting didn't work. The system log tells us why:
Jan 4 22:11:27 home vde_switch[31504]: Error in pidfile creation: File exists
So, VDE has a bug in its pidfile creation. There are two ways how one can deal with this: either tell systemd to remove the PID file before starting vde_switch, or drop the PID file altogether (because vde_switch has exactly one process, there can be no confusion which process is the main one). Both ways work. Here is how to implement the first alternative:
[Unit] Description=Virtual Distributed Ethernet After=syslog.target [Service] Type=forking PIDFile=/var/run/vde.pid # Note the -f: don't fail if there is no PID file ExecStartPre=/bin/rm -f /var/run/vde.pid ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \ --dirmode 0750 --group qemu \ --daemon --pidfile /var/run/vde.pid Restart=on-abort [Install] WantedBy=multi-user.target
And here is the second alternative:
[Unit] Description=Virtual Distributed Ethernet After=syslog.target [Service] Type=forking ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \ --dirmode 0750 --group qemu \ --daemon Restart=on-abort [Install] WantedBy=multi-user.target
Configuration
Many initscripts come with configuration files that allow the user to customize how the daemon is started. For example, some users may want to pass the --hub argument to vde_switch, and others may want to enable the management socket. So, in Gentoo, the traditional configuration file for the initscript looks like this:
# load the tun module
VDE_MODPROBE_TUN="yes"
# virtual tap networking device to be used for vde
VDE_TAP="tap0"
# mode and group for the socket
VDE_SOCK_CHMOD="770"
VDE_SOCK_CHOWN=":qemu"
# This is the actual options string passed to VDE.
# Change this at your own risk.
VDE_OPTS="--tap ${VDE_TAP} -daemon"
Systemd service files typically use the EnvironmentFile key to provide users with a file where they can put their preferences regarding the service.
Traditional initscripts source their configuration files. Thus, any syntax construction supported by /bin/sh will work in the configuration file. In the example above, we see comments, assignment of values to variables, and reusing the values in later assignments. Systemd uses a different syntax from bash, so please resist the temptation to reuse the same configuration file for the traditional initscript and the service file. Resist even though some service files in the Gentoo systemd overlay do use the same configuration files as the corresponding traditional initscripts -- they are just buggy. Let me explain this in more detail.
Of course, the configuration file example above is not suitable for systemd because variable interpolation is not supported in systemd environment files. But let's suppose that we want to invent something that is suitable both as a bash script fragment and a systemd environment file, and still allows the user to configure vde_switch according to his wishes.
Let's focus on the VDE_OPTS variable only, as it is the only thing that matters. Indeed, module loading can be done directly by systemd (man modules-load.d), and the options related to the socket group and octal permissions can be expressed using vde_switch command line options, as illustrated in the examples above. Since the value of the VDE_OPTS variable can contain spaces, we have to quote it if we want bash to be able to understand what we mean:
VDE_OPTS="--tap tap0 --mode 0660 --dirmode 0750 --group qemu"
Without quotes, bash would interpret this as follows: with the variable VDE_OPTS that has value "--tap" in the environment, start the "tap0" process and pass 6 parameters to it. So, quotes are essential here.
Let's try to use this variable from the service file. We want it to be split by the spaces, so that each part becomes a separate vde_switch parameter. So, according to the systemd.service manpage, we have to use the $VDE_OPTS form, not ${VDE_OPTS}. So here is what we have:
[Unit] Description=Virtual Distributed Ethernet After=syslog.target [Service] Type=forking EnvironmentFile=/etc/conf.d/vde2 ExecStart=/usr/bin/vde_switch --daemon $VDE_OPTS Restart=on-abort [Install] WantedBy=multi-user.target
Result: no "tap0" interface and wrong permissions on the control socket. This happened because quotes play a special role in systemd environment files, different from their role in bash scripts. For systemd, they mean that the spaces inside them should not be treated as argument separators. So, all the arguments in $VDE_OPTS were passed to vde_switch as one long argument. No surprise that it didn't work.
In fact, the service file is correct. Its configuration just can't be made compatible with bash, because it is in a different language. The service works with the following configuration file (alas, incompatible with bash):
VDE_OPTS=--tap tap0 --mode 0660 --dirmode 0750 --group qemu
Conclusion
Let's hope that all of the above will get you started writing your own systemd service files. Since there are many initscripts in Gentoo still not converted, the project needs your help.
Thanks
The following people from #systemd IRC channel on freenode provided me with valuable support: MK_FG, zdzichuBG, miti1.
Friday, November 12, 2010
Minimizing server downtime
Suppose that you are responsible for a typical LAMP server running Debian. The goal is to have the needed security updates and minimize downtime. Unfortunately, due to installation of security updates, services need to be restarted, and sometimes (after kernel updates) even a reboot is required. Here are some tips that will help you shrink the time of service unavailability.
- Upgrade your key services (apache, php and mysql) separately from the rest of software. Reason: if you just run "apt-get dist-upgrade" to upgrade everything, then your MySQL server will be stopped at the beginning of the whole upgrade and started only at the end, after unpacking all other packages. So, run "apt-get install mysql-server" before "apt-get dist-upgrade" to minimize the delay.
- If you know that kexec works on your hardware, use it: apt-get install kexec-tools. This way, you will avoid the long time (sometimes more than a minute) while your server shows useless BIOS screens. Instead of rebooting to the BIOS, the old kernel will load the new kernel directly and pass control to it, thus avoiding the delay.
Saturday, October 23, 2010
DNS servers in resolv.conf
Yesterday I found a problem with my Linode VPS. When running "aptitude update", it complained about not being able to resolve the name for security.debian.org (a server from where Debian distributes security updates).
Here is the /etc/resolv.conf file from that VPS, generated by dhclient:
It looks like the 109.74.194.20 server is overloaded and returns a SERVFAIL ("server failure - The name server was unable to process this query due to a problem with the name server") answer from time to time. Unfortunately, Debian has a bug in its version of libc that prevents the resolver from trying the other (working) servers if the first one returns an answer indicating a temporary failure.
The bug does not exist in glibc-2.12.1 on my Gentoo box at home. However, if the server does not respond at all, the Gentoo resolver spends 15 seconds before trying the next one, which is also not nice.
To work around the bug, you can run your own DNS server on 127.0.0.1 that forwards all queries to the official servers, and put 127.0.0.1 as the only nameserver in /etc/resolv.conf.
Here is the /etc/resolv.conf file from that VPS, generated by dhclient:
domain members.linode.com search members.linode.com nameserver 109.74.194.20 nameserver 109.74.192.20 nameserver 109.74.193.20
It looks like the 109.74.194.20 server is overloaded and returns a SERVFAIL ("server failure - The name server was unable to process this query due to a problem with the name server") answer from time to time. Unfortunately, Debian has a bug in its version of libc that prevents the resolver from trying the other (working) servers if the first one returns an answer indicating a temporary failure.
The bug does not exist in glibc-2.12.1 on my Gentoo box at home. However, if the server does not respond at all, the Gentoo resolver spends 15 seconds before trying the next one, which is also not nice.
To work around the bug, you can run your own DNS server on 127.0.0.1 that forwards all queries to the official servers, and put 127.0.0.1 as the only nameserver in /etc/resolv.conf.
Thursday, August 5, 2010
Floating-point fields in MySQL
It is a well-known fact that floating-point values should not be compared for equality, due to rounding problems. MySQL puts an interesting twist to it, as demonstrated on the linux.org.ru forum. To see the strange effect, run the following SQL statements:
I.e., it looks like MySQL successfully stores the value, but cannot find it later.
To see why it happens, recall that 0.3 cannot be represented exactly in binary form. It is a recurring binary fraction 0b0.0100110011... that has to be truncated somewhere when stored by a computer. When stored as a float data type in the table, 24 significant binary digits are retained. However, when 0.3 is mentioned directly in the query, it is treated as a double-precision number, and 53 significant bits are kept. The truncation thus occurs in different places, and the resulting numbers are just different:
MySQL documentation contains more information on problems with floating-point values.
mysql> create table test (field float);
mysql> insert into test(field) values(0.3);
mysql> select * from test;
+-------+
| field |
+-------+
| 0.3 |
+-------+
1 row in set (0.00 sec)
mysql> select * from test where field = 0.3;
Empty set (0.00 sec)
I.e., it looks like MySQL successfully stores the value, but cannot find it later.
To see why it happens, recall that 0.3 cannot be represented exactly in binary form. It is a recurring binary fraction 0b0.0100110011... that has to be truncated somewhere when stored by a computer. When stored as a float data type in the table, 24 significant binary digits are retained. However, when 0.3 is mentioned directly in the query, it is treated as a double-precision number, and 53 significant bits are kept. The truncation thus occurs in different places, and the resulting numbers are just different:
mysql> select cast(field as decimal(15,15)) from test; +-------------------------------+ | cast(field as decimal(15,15)) | +-------------------------------+ | 0.300000011920929 | +-------------------------------+ 1 row in set (0.00 sec) mysql> select cast(0.3 as decimal(15,15)); +-----------------------------+ | cast(0.3 as decimal(15,15)) | +-----------------------------+ | 0.300000000000000 | +-----------------------------+ 1 row in set (0.00 sec)
MySQL documentation contains more information on problems with floating-point values.
Saturday, May 29, 2010
When linear search is faster than binary
It is often stated that a binary search algorithm performs better than linear search. Indeed, let's assume that the goal is to find the index of the largest integer in the sorted zero-based array A that is less than or equal to the given integer X. For simplicity, let's also assume that the size of array is N = 2K, that for all possible inputs A[0] <= X and that one can also put a sentinel value after the end, so that X < A[N] for all possible inputs X. E.g., if it is known that X always fits in 16 bits, one can put 65536 as the sentinel.
The assumptions above are true for the entropy decoder found in the Monkey's Audio codec. In that case, N = 64, X is unsigned and always fits in 16 bits, A[0] = 0, A[64] = 65536.
This is linear search:
This is binary search:
If the size of array to be searched is N, then the average case complexity is commonly stated to be O(log(N)) for binary search and O(N) for linear search. So, for the Monkey's Audio case presented above, one would expect, on average, 32 loop iterations for the linear search algorithm and 6 iterations for the binary search.
However, there is one key fact that makes the estimate above invalid. Such simple averaging is valid only if each result appears with the same probability. For Monkey's Audio entropy decoder, this is not the case.
In fact, out of the 64 possible results, "0" appears with 31% probability, and the total probability of the first 6 results is 87%. So, it is quite natural that linear search is faster than binary in this case, because it often stops very early.
The assumptions above are true for the entropy decoder found in the Monkey's Audio codec. In that case, N = 64, X is unsigned and always fits in 16 bits, A[0] = 0, A[64] = 65536.
This is linear search:
A[N] = sentinel;
index = 0;
while (A[index + 1] <= X)
++index;
This is binary search:
int bit = N >> 1; /* N is a power of two */
index = 0;
while (bit != 0) {
if (A[index + bit] <= X)
index += bit;
bit >>= 1;
}
If the size of array to be searched is N, then the average case complexity is commonly stated to be O(log(N)) for binary search and O(N) for linear search. So, for the Monkey's Audio case presented above, one would expect, on average, 32 loop iterations for the linear search algorithm and 6 iterations for the binary search.
However, there is one key fact that makes the estimate above invalid. Such simple averaging is valid only if each result appears with the same probability. For Monkey's Audio entropy decoder, this is not the case.
In fact, out of the 64 possible results, "0" appears with 31% probability, and the total probability of the first 6 results is 87%. So, it is quite natural that linear search is faster than binary in this case, because it often stops very early.
Subscribe to:
Posts (Atom)