Tuesday, January 4, 2011

Writing systemd service files

Today I tried to convert a Gentoo initscript for VDE to a systemd service file. This post documents the required steps. VDE has been chosen as an example because it is a simple daemon (commonly used for communication between several instances of QEMU and the host) that illustrates the matter well.

Background


For those who don't know, systemd is a next-generation replacement for /sbin/init written by Lennart Poettering. While systemd supports traditional initscripts, service files are its native form of configuration. Service files have a structure similar to that of desktop files or Windows ini files. Their syntax is well documented in the manual pages.

Initscripts typically contain boilerplate code that checks whether the daemon is already running or figures out which process to kill when the daemon has to be stopped. Tools like start-stop-daemon help, but systemd reduces the syntax overhead to the minimum. This simplification occurs because service files specify what should be done, not how it should be done. I.e., unlike initscripts, they follow the declarative style and are not programs.

Simple services


Here is the simplest possible service file that starts VDE with options that work for me on my computer. Save it as /etc/systemd/system/vde.service:

[Unit]
Description=Virtual Distributed Ethernet

[Service]
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu

[Install]
WantedBy=multi-user.target


Note the difference from a traditional init script: for simplicity, vde_switch is started in such a way that it doesn't become a daemon. In fact, it is possible to start real daemons that fork, you just have to tell systemd about that:

[Unit]
Description=Virtual Distributed Ethernet

[Service]
Type=forking
# The PID file is optional, but recommended in the manpage
# "so that systemd can identify the main process of the daemon"
PIDFile=/var/run/vde.pid
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu \
 --daemon --pidfile /var/run/vde.pid

[Install]
WantedBy=multi-user.target


The difference is in the way the dependencies are handled. If some other services depend on vde.service, then, in the first example, systemd will be able to run them as soon as it starts vde_switch. In the second example, systemd will wait until vde_switch forks. The difference matters, because vde_switch creates its control socket after starting, but before forking. So, in the first example, there is some chance that systemd will start something that tries to connect to the socket before vde_switch creates it.

Automatic restarts


Let's also add the proper dependency on the system logger and tell systemd to restart vde_switch if it crashes due to an uncaught signal (although it never happened to me):

[Unit]
Description=Virtual Distributed Ethernet
After=syslog.target

[Service]
Type=forking
PIDFile=/var/run/vde.pid
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu \
 --daemon --pidfile /var/run/vde.pid
Restart=on-abort

[Install]
WantedBy=multi-user.target

Now try to start and crash the daemon:

home ~ # systemctl start vde.service
home ~ # systemctl status vde.service
vde.service - Virtual Distributed Ethernet
   Loaded: loaded (/etc/systemd/system/vde.service)
   Active: active (running) since Tue, 04 Jan 2011 22:08:10 +0500;
15s ago
  Process: 31434 (/usr/bin/vde_switch --tap tap0...,
code=exited, status=0/SUCCESS)
 Main PID: 31435 (vde_switch)
   CGroup: name=systemd:/system/vde.service
    └ 31435 /usr/bin/vde_switch --tap tap0...
home ~ # kill -SEGV 31435
home ~ # systemctl status vde.service
vde.service - Virtual Distributed Ethernet
   Loaded: loaded (/etc/systemd/system/vde.service)
   Active: failed since Tue, 04 Jan 2011 22:11:27 +0500;
4s ago
  Process: 31503 (/usr/bin/vde_switch --tap tap0...,
code=exited, status=0/SUCCESS)
 Main PID: 31504 (code=exited, status=1/FAILURE)
   CGroup: name=systemd:/system/vde.service

I.e., restarting didn't work. The system log tells us why:

Jan  4 22:11:27 home vde_switch[31504]: Error in pidfile
creation: File exists

So, VDE has a bug in its pidfile creation. There are two ways how one can deal with this: either tell systemd to remove the PID file before starting vde_switch, or drop the PID file altogether (because vde_switch has exactly one process, there can be no confusion which process is the main one). Both ways work. Here is how to implement the first alternative:

[Unit]
Description=Virtual Distributed Ethernet
After=syslog.target

[Service]
Type=forking
PIDFile=/var/run/vde.pid
# Note the -f: don't fail if there is no PID file
ExecStartPre=/bin/rm -f /var/run/vde.pid
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu \
 --daemon --pidfile /var/run/vde.pid
Restart=on-abort

[Install]
WantedBy=multi-user.target

And here is the second alternative:
[Unit]
Description=Virtual Distributed Ethernet
After=syslog.target

[Service]
Type=forking
ExecStart=/usr/bin/vde_switch --tap tap0 --mode 0660 \
 --dirmode 0750 --group qemu \
 --daemon
Restart=on-abort

[Install]
WantedBy=multi-user.target

Configuration


Many initscripts come with configuration files that allow the user to customize how the daemon is started. For example, some users may want to pass the --hub argument to vde_switch, and others may want to enable the management socket. So, in Gentoo, the traditional configuration file for the initscript looks like this:

# load the tun module
VDE_MODPROBE_TUN="yes"
# virtual tap networking device to be used for vde
VDE_TAP="tap0"
# mode and group for the socket
VDE_SOCK_CHMOD="770"
VDE_SOCK_CHOWN=":qemu"

# This is the actual options string passed to VDE.
# Change this at your own risk.
VDE_OPTS="--tap ${VDE_TAP} -daemon"

Systemd service files typically use the EnvironmentFile key to provide users with a file where they can put their preferences regarding the service.

Traditional initscripts source their configuration files. Thus, any syntax construction supported by /bin/sh will work in the configuration file. In the example above, we see comments, assignment of values to variables, and reusing the values in later assignments. Systemd uses a different syntax from bash, so please resist the temptation to reuse the same configuration file for the traditional initscript and the service file. Resist even though some service files in the Gentoo systemd overlay do use the same configuration files as the corresponding traditional initscripts -- they are just buggy. Let me explain this in more detail.

Of course, the configuration file example above is not suitable for systemd because variable interpolation is not supported in systemd environment files. But let's suppose that we want to invent something that is suitable both as a bash script fragment and a systemd environment file, and still allows the user to configure vde_switch according to his wishes.

Let's focus on the VDE_OPTS variable only, as it is the only thing that matters. Indeed, module loading can be done directly by systemd (man modules-load.d), and the options related to the socket group and octal permissions can be expressed using vde_switch command line options, as illustrated in the examples above. Since the value of the VDE_OPTS variable can contain spaces, we have to quote it if we want bash to be able to understand what we mean:

VDE_OPTS="--tap tap0 --mode 0660 --dirmode 0750 --group qemu"

Without quotes, bash would interpret this as follows: with the variable VDE_OPTS that has value "--tap" in the environment, start the "tap0" process and pass 6 parameters to it. So, quotes are essential here.

Let's try to use this variable from the service file. We want it to be split by the spaces, so that each part becomes a separate vde_switch parameter. So, according to the systemd.service manpage, we have to use the $VDE_OPTS form, not ${VDE_OPTS}. So here is what we have:

[Unit]
Description=Virtual Distributed Ethernet
After=syslog.target

[Service]
Type=forking
EnvironmentFile=/etc/conf.d/vde2
ExecStart=/usr/bin/vde_switch --daemon $VDE_OPTS
Restart=on-abort

[Install]
WantedBy=multi-user.target

Result: no "tap0" interface and wrong permissions on the control socket. This happened because quotes play a special role in systemd environment files, different from their role in bash scripts. For systemd, they mean that the spaces inside them should not be treated as argument separators. So, all the arguments in $VDE_OPTS were passed to vde_switch as one long argument. No surprise that it didn't work.

In fact, the service file is correct. Its configuration just can't be made compatible with bash, because it is in a different language. The service works with the following configuration file (alas, incompatible with bash):

VDE_OPTS=--tap tap0 --mode 0660 --dirmode 0750 --group qemu

Conclusion


Let's hope that all of the above will get you started writing your own systemd service files. Since there are many initscripts in Gentoo still not converted, the project needs your help.

Thanks


The following people from #systemd IRC channel on freenode provided me with valuable support: MK_FG, zdzichuBG, miti1.

13 comments:

Alexander E. Patrakov said...

The quote behaviour has been changed in systemd-16. Here os a git commit: http://cgit.freedesktop.org/systemd/commit/?id=5f7c426e2a7f72c473f98be9978d243db79d8910

So my words about impossibility to have the same file with options parsed correctly in both bash and systemd may be no longer correct. I didn't check.

dave@practicalclouds.com said...

Thanks for the helpful article! I found another issue, where my scripts I wanted to run at shutdown were not being called before "halt", "reboot" or "poweroff", "init 0" etc..

It took me a while to find a solution, and so I have written about it here http://www.practicalclouds.com/content/blog/1/dave-mccormick/2012-02-27/why-do-my-systemd-shutdown-scripts-not-run

regards


Dave

kettlewell said...

Alexander -

Thanks for the step by step examples, I'd been looking for something like this.

I was wondering why the /etc/init.d/* directory was getting slim, and found that Fedora was making the switch to systemd.

I'm not sure of why, but some of the sysV init scripts aren't starting automatically after boot up, so I thought I'd attempt to create a .service file, and see if it fixes the problem.

Your example will serve me well.

Thanks a bunch.

Matt

Peter said...

Thanks!

Samuel Alfonso Velázquez Díaz said...

Thanks for the example!

Mark said...

This is very useful. There is a mistake in your example for EnvironmentFile. You need to dereference variables with curly braces, so ${VDE_OPTS} and not $VDE_OPTS. See discussion here

https://ask.fedoraproject.org/en/question/10474/why-systemd-is-not-loading-environment-file/

Alexander E. Patrakov said...

There is no mistake. ${VDE_OPTS} means that the string will not be split by spaces into separate arguments, and we do want splitting here.

Old guy said...

Having read through the man page, my view is that that we have a poor design decision by the systemd developers: to use the same variable syntax as another well-established language (Bourne shell with Dollar symbol, braces etc) but change the semantics. As this helpful blog already shows, it guarantees confusion. The English idiom "Don't upset the apple cart" covers this (The doubles quotes holding the idiom together).

Unknown said...

Hi,

nice article, though I would like to read something about getting systemd to use a specific service file. This is I encountered the following behavior:
I wrote a service file, but had a syntax error in it, so after doing "systemctl daemon-reload", "systemctl status myservice" told me about the error.
So far so good, but after correcting the error and issuing another daemon-reload nothing changed... after quite a while of trial and error , I completely removed the service file an triggered the daemon-reload, awaiting that systemd would tell me it doesn't know about the service...
But wrong... it kept issuing the initial error... so it must be caching these files somewhere
Thank god the designers of the distribution (redhat 7) thought it would be a good idea to remove these directories from the list of directories indexed by locate, so I couldn't find them without doing a "find" which would take way too long, since I'm on a server with millions of smalls and about 1 TB storage...
So do you have any clue how to get this great piece of software to accept my corrected service file? (Sorry for that irony, I really try to find the positive aspects of systemd but everytime I do, it suprises my again with some of this completely irrational behaviors)

Alexander E. Patrakov said...

For the record, now systemd parses quoted lines in environment files properly. The problem described in the post is obsolete. And anyway, environment files are now discouraged, configuration (if any) should be placed in drop-in service files.

Anonymous said...

Can you please clarify when do we need to keep RemainAfterExit=True while creating new services?

JohnSmith said...

Nice blog post, Thanks a lot providing such a clear explanation.

Ricardo Martins said...

Thank you very much for sharing.
It helped a lot here.