modifying systemd unit files

This is mainly for my reference, and will get updated from time to time. More systemd stuff via the systemd tag.

Introduction

On systems with init.d start scripts, if it was necessary to modify a boot script, updating the package would back out the change.

Systemd provides a number of ways to change the behaviour of systemd unit files, making modifications to the original file explicitly bad practise.

I’ll be looking at service unit files, but I assume the principles apply to timers, sockets, targets etc. There maybe multiple units with the same unqualified name (eg: lvm2-lvmetad is both a .service and .socket on RHEL7) so I think it’s best practise to qualify unit names.

Loading changes to unit files without rebooting.

refs: redhat.

Either of the following, and then restart the modified service.

systemctl daemon-reload
init q                     # see init --help, and see qualification below.

[..]

systemctl restart some.service

Using systemctl edit loads the changes into systemd. ‘edit’ takes different forms depending on how you want to modify the configuration.

init -q doesn’t seem to be reliably supported;  this despite showing in the usage statement for init.

# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core) 
# init -q
init: invalid option -- 'q'

/run

refs: systemctl man page.

This is referred to in the man pages; you can make and load non persistent unit file changes. The easiest way to do this is probably with edit, as I’ve not seen specifics on how to do this any other way. Edit ought to load the modifications for you as well.

systemctl edit --runtime some.service

This will only work to over-ride configuration in /usr/lib/systemd/system – changes in /etc take a higher precedent (/etc -> /run -> /usr)

If the file is empty after editing, the change is cancelled.

So, this works nicely as a way to experiment with a change, and if it all goes Pete Tong, you just reboot. Yay.

example

# export EDITOR=$(which vi)
# systemctl edit --runtime lvm2-lvmetad.service
[..]
# cat /run/systemd/system/lvm2-lvmetad.service.d/override.conf
[Service]
ExecStart=/usr/sbin/lvmetad -f -l all
# systemctl status lvm2-lvmetad.service
 lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: error (Reason: Invalid argument)
  Drop-In: /run/systemd/system/lvm2-lvmetad.service.d
           └─override.conf
   Active: active (running) since Wed 2019-04-24 08:40:38 BST; 1h 12min ago
     Docs: man:lvmetad(8)
 Main PID: 1467 (lvmetad)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1467 /usr/sbin/lvmetad -f

Apr 24 08:40:38 copernicus systemd[1]: Started LVM2 metadata daemon.
Apr 24 09:51:08 copernicus systemd[1]: lvm2-lvmetad.service has more than one 
                                       ExecStart= setting, which is only allowed 
                                       for Type=oneshot services. Refusing.

So, it’s useful doing this stuff on the fly in /run first. Owing to the error I couldn’t then do this though:

# systemctl edit --runtime lvm2-lvmetad.service 
Unit lvm2-lvmetad.service is not loaded: Invalid argument

So, I edited the /run file and reloaded systemd.

systemd[1]: Current command vanished from the unit file, execution of the command list won't be resumed.

Otherwise it’s happy to open the file for you.  I imagine it only supports override.conf this way, if you create lots of .conf files (see example below) it’s not going to know which one to edit.

identifying that changes have been made

refs: man pages as specified, redhat, archlinux.

Query the specific properties you’re interested in:

systemctl daemon-reload  # just in case it's not obvious that you need to do this!
systemctl show lvm2-lvmetad.service --property=ExecStart

Show the unit configuration ..

systemctl cat lvm2-lvmetad.service

Check the whole system with

systemd-delta

This will mark up changes in a number of ways;  and you can ask it just to show a specific sort of change.  From its man page:

masked
    Show masked files

equivalent
    Show overridden files that while overridden, do not differ in content.

redirected
    Show files that are redirected to another.

overridden
    Show overridden, and changed files.

extended
    Show *.conf files in drop-in directories for units.

unchanged
    Show unmodified files too.

example

# systemctl show lvm2-lvmetad.service --property=ExecStart
ExecStart={ path=/usr/sbin/lvmetad ; argv[]=/usr/sbin/lvmetad -f -l all ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; 

# systemctl cat lvm2-lvmetad.service
# /usr/lib/systemd/system/lvm2-lvmetad.service
[Unit]
Description=LVM2 metadata daemon
Documentation=man:lvmetad(8)
Requires=lvm2-lvmetad.socket
After=lvm2-lvmetad.socket
DefaultDependencies=no
Conflicts=shutdown.target

[Service]
Type=simple
NonBlocking=true
ExecStart=/usr/sbin/lvmetad -f
Environment=SD_ACTIVATION=1
Restart=on-abort
PIDFile=/run/lvmetad.pid

# /run/systemd/system/lvm2-lvmetad.service.d/override.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/lvmetad -f -l all

Systemd makes heavy use of /run for transient configuration, userspace processes and so on.

# systemd-delta | egrep -v '.scope|.slice'
[EQUIVALENT] /etc/systemd/system/default.target → /usr/lib/systemd/system/default.target
[EXTENDED]   /usr/lib/systemd/system/lvm2-lvmetad.service → /run/systemd/system/lvm2-lvmetad.service.d/override.conf

34 overridden configuration files found.

/etc/systemd/system/(unit).d/

refs: redhat, archlinux.

Used to persistently extend/modify the existing configuration while referring to the original. This allows changes with package updates to be applied.

“Drop-in files” are created in these directories with the .conf extension.

mkdir /etc/systemd/system/(name).service.d
touch /etc/systemd/system/(name).service.d/override.conf

The contents of the file should be consistent with the unit file structure. To modify a given property, it must be specified in the same section within the drop-in file as it would be if you were changing the unit file.

For example, if you wanted to add command line arguments to lvm2-lvmetad.

mkdir /etc/systemd/system/lvm2-lvmetad.service.d
cat > /etc/systemd/system/lvm2-lvmetad.service.d/params.conf <<EOF
[Service]
ExecStart=
ExecStart=/usr/sbin/lvmetad -f -l all
EOF

(You have to empty ExecStart first – see just below for more, plus – under /run – what happens if you don’t.)

Other changes that you might need to make are adding dependencies, or reconfiguring how systemd restarts the service if it fails – such as adding that feature and / or specifying a different delay before performing the restart.

Persistant changes can also be made with:

systemctl edit some.service

This creates the directory and a file called override.conf and loads the unit file when saved. Restart the affected service to be sure it has changed; see below.

constraints based on key type

Some keys can’t be modified using drop-in files, such as Conflicts= – it seems drop-ins cannot remove dependencies.

Under some circumstances, you may need to empty a key before setting it, such as when a value can be set multiple times. Ref.

Or, when systemd gets confused, for example ExecStart – which can be set multiple times, just not always – see /run above.

[Service]
ExecStart=
ExecStart=/some/command

example

Pretty much works as expected.

# systemctl cat lvm2-lvmetad.service
# /usr/lib/systemd/system/lvm2-lvmetad.service
[Unit]
Description=LVM2 metadata daemon
Documentation=man:lvmetad(8)
Requires=lvm2-lvmetad.socket
After=lvm2-lvmetad.socket
DefaultDependencies=no
Conflicts=shutdown.target

[Service]
Type=simple
NonBlocking=true
ExecStart=/usr/sbin/lvmetad -f
Environment=SD_ACTIVATION=1
Restart=on-abort
PIDFile=/run/lvmetad.pid

# /etc/systemd/system/lvm2-lvmetad.service.d/params.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/lvmetad -f -l all

Subject to the priorities being understood (/etc over-writes /run, both of which over-write /usr) it copes with multiple changes at different levels:

# systemctl status lvm2-lvmetad
 lvm2-lvmetad.service - LVM2 metadata daemon - with modifications
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; static; vendor preset: enabled)
  Drop-In: /etc/systemd/system/lvm2-lvmetad.service.d
           └─doc.conf
        /run/systemd/system/lvm2-lvmetad.service.d
           └─override.conf
        /etc/systemd/system/lvm2-lvmetad.service.d
           └─params.conf
   Active: active (running) since Wed 2019-04-24 09:25:22 BST; 1h 38min ago
     Docs: man:lvmetad(8)
           http://man7.org/linux/man-pages/man8/lvmetad.8.html
 Main PID: 1470 (lvmetad)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─1470 /usr/sbin/lvmetad -f


# systemctl cat lvm2-lvmetad
# /usr/lib/systemd/system/lvm2-lvmetad.service
[Unit]
Description=LVM2 metadata daemon
Documentation=man:lvmetad(8)
Requires=lvm2-lvmetad.socket
After=lvm2-lvmetad.socket
DefaultDependencies=no
Conflicts=shutdown.target

[Service]
Type=simple
NonBlocking=true
ExecStart=/usr/sbin/lvmetad -f
Environment=SD_ACTIVATION=1
Restart=on-abort
PIDFile=/run/lvmetad.pid

# /etc/systemd/system/lvm2-lvmetad.service.d/doc.conf
[Unit]
Documentation='http://man7.org/linux/man-pages/man8/lvmetad.8.html'
# /run/systemd/system/lvm2-lvmetad.service.d/override.conf
[Unit]
Description=LVM2 metadata daemon - with modifications
# /etc/systemd/system/lvm2-lvmetad.service.d/params.conf
[Service]
ExecStart=
ExecStart=/usr/sbin/lvmetad -f -l all

restarting the service/unit

Systemd wasn’t interested in applying my ExecStart change to a running service.

Current command vanished from the unit file, execution of the command list won't be resumed.

YMMV: other changes such as timeouts, dependencies which are internal to systemd might get applied on the fly.

Restarting the service is going to be the sure fire way to know, and when changing things material to booting the system, testing it somewhere with a reboot is always a good idea.

/etc/systemd/system

refs: redhat, archlinux.

The configuration is copied here from /usr/lib/systemd/system/.  It replaces the original file entirely.

cp /usr/lib/systemd/system/some.service /etc/systemd/system/some.service

I found references to another way to load unit changes in this case:

systemctl reenable some.service

Or, the one-stop-shop approach – create the copy, present it for editing, and do a reload to apply the changes, at least in systemd.

systemctl edit --full some.service

modifying systemd properties of init.d managed services

refs: redhat .

You might want to change dependencies or timeouts.

  • Don’t modify the script in /etc/rc.d/init.d.
  • Follow the drop-in file approach detailed under the (unit).d section above.
  • Ensure the directory name matches how systemd knows this init.d service. Redhat provides the example of init.d/network – systemd knows this as network.service

This I can see as useful, as an in-house written init.d script might not have the comments set to provide the write dependencies, and so it’ll get stopped and started fairly arbitrarily.

There’s almost a case to not attempting to create a unit file for your legacy start/stop script – just use drop in files to populate the systemd properties.

set-property

Individual properties can be queried (as shown above) and in some cases set.  The man page for systemctl implies it’ll do it persistently (/etc) or transiently (/run)

Not all properties may be changed at runtime, but many resource control
settings (primarily those in systemd.resource-control(5)) may. The changes
are applied instantly, and stored on disk for future boots, unless –runtime
is passed, in which case the settings only apply until the next reboot.

# systemctl show lvm2-lvmetad --property=ExecStart
ExecStart={ path=/usr/sbin/lvmetad ; argv[]=/usr/sbin/lvmetad -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(nul
# systemctl set-property lvm2-lvmetad.service ExecStart='/usr/sbin/lvmetad -f -l all'
Unknown assignment ExecStart=/usr/sbin/lvmetad -f -l all.

So, that’d be a case of one that doesn’t relate to resource control.  I’d be interested to see what form the drop-in files take, but not interested enough that I have a worked example.

Rolling back to the vendor configuration

refs: archlinux.

systemctl revert some.service

This isn’t documented on RHEL7 (systemd 219) so would seem to be a later feature.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s