Introduction: services that depend on NFS

If a linux server boots and a block device isn’t available, it’ll drop into some sort of single-user mode. This was the way with init, and is the way with systemd.

NFS, not so much. In my experience, systemd will start services implicitly dependent on an NFS mount the moment the network comes up. The mount might be there, it might be there in a little while, it might never turn up. It’s not considered a show stopper like block devices are.

Systemd fabricates services for mounts; stumbling only slightly if they have hyphens in them:

var-opt-gitlab-git\x2ddata-repositories.mount   loaded active mounted   /var/opt/gitlab/git-data/repositories
var-opt-gitlab.mount                            loaded active mounted   /var/opt/gitlab

So, it should be possible to use before, after, requires, wants and friends to ensure that a service doesn’t start without its dependencies.

Two ways:

1. Add the relationship on the mount unit.
2. Add the relationship on the service unit.

There’s a detail in the systemd documentation that’s of particular interest. When looking at before, after, requires and other unit configuration settings, some:

may be specified more than once

This is documented for before, after, and wants, implied for requires, and unknown for RequiresMountsFor.

• If true, then I believe you could supply multiple modifying unit files with different services (or mounts) and systemd will merge them all into a list.
• If false, you can only supply the list once, which is less flexible.

I’m doing this via a looping class in Puppet: so the ability just to create separate unit modification files for each dependency would be nice.

How

• More about modifying unit files in my earlier post.
• File can be dropped into /etc/systemd/system/foo.service.d or /etc/systemd/system/foo.mount.d to mask, modify, extend units, assuming they’re configured in either /run/systemd/system (volatile) or /lib/systemd/system.
• I’m not clear to what extent units configured in /etc can also be extended via /etc.  You definitely cannot mask them.

The Right Relationship is Everything

• Use requires and after if the dependency must be enforced, which is the point.

Requires= [..] If one of the other units fails to activate, and an ordering dependency After= on the failing unit is set, this unit will not be started.

Via the Mount Unit

• There’s a RequiredBy Install option; the mirror image of Requires.
• RequiredBy goes under the ‘Install’ section and only works when enabling or disabling the service (mount unit). I want to just HUP systemd to implement the dependency. It’s kind of inflexible that I might need to persuade systemd that the fstab entry has been added.
• How it works in practise is it creates a symlink to modify the service unit.
  • It creates a .requires directory and then links the Mount unit into it.
  • This is how boot milestones are created, such as /lib/systemd/system/multi-user.target.wants.
  • I could just implement the symlink and directory and then nudge systemd.
• I still want to do some of the heavy lifting on the mount unit.
  • I’d specify the mount as before the service
  • That ought to be identical to the service being after the mount.
  • The docs don’t say Before; they say After:

If one of the other units fails to activate, and an ordering dependency After= on the failing unit is set, this unit will not be started

Conclusion:

• I really wanted to do this to one unit file only
• Actually, it’s probably nicer to do it on the service unit.
• It’s less likely that multiple services will depend on a mount, and very likely that a service will depend on multiple mounts, and achieving this is an outstanding question.

Via the Service Unit

• There’s a mount specific option RequiresMountsFor available which takes the path names, and implements Requires + After.  Given that this is exactly what I’m trying to do, seems rude not to.
• Don’t know when RequiresMountsFor was introduced, but it works on Centos (RHEL) 7.7
• It’s more than possible that a service might depend on multiple mounts. I’ve not tested whether I can supply multiple additional configuration files.

Testing

• Implemented as follows:

# /run/systemd/system/puppettestsvc.service
# puppet managed by profile::test::systemd_svc
[Unit]
Description=Test systemd service puppettestsvc
After=network.target
Requires=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/puppet_test_service.sh start
ExecStop=/usr/local/bin/puppet_test_service.sh stop
TimeoutStopSec=5s
Restart=no

# /etc/systemd/system/puppettestsvc.service.d/puppet-fs_mount-tmpscratch.conf
# managed by puppet profile::fs_mount
[Unit]
RequiresMountsFor=/var/tmp/scratch

• service online, filesystem mounted. Mount unit taken offline; service is stopped first.

# systemctl stop var-tmp-scratch.mount
# journactl -f
Dec 29 18:04:32 endeavour systemd[1]: Stopping Test systemd service puppettestsvc...
Dec 29 18:04:35 endeavour systemd[1]: Stopped Test systemd service puppettestsvc.
Dec 29 18:04:35 endeavour systemd[1]: Unmounting /var/tmp/scratch...
Dec 29 18:04:35 endeavour systemd[1]: Unmounted /var/tmp/scratch.

• service offline, filesystem not mounted. Service started; systemd automatically brings the mount on line.

# systemctl start puppettestsvc
# journactl -f
Dec 29 17:59:35 endeavour systemd[1]: Mounting /var/tmp/scratch...
Dec 29 17:59:35 endeavour systemd[1]: Mounted /var/tmp/scratch.
Dec 29 17:59:35 endeavour systemd[1]: Started Test systemd service puppettestsvc.

• service online, filesystem mounted. Filesystem unmounted with umount.
  • Service is not stopped.
  • Possibly quite nice.  Allows for administrative intervention while keeping the service online.
• Service and mount offline. Mount definition changed so it’s not valid.
  • Service started.
  • Mount and service brought online.
  • Ah hah!  ‘Fixed’ systemd and then it worked correctly:

Warning: var-tmp-scratch.mount changed on disk. Run 'systemctl daemon-reload' to reload units.

# systemctl daemon-reload
# systemctl status var-tmp-scratch.mount   # inactive
# systemctl status puppettestsvc           # inactive
# systemctl start puppettestsvc
A dependency job for puppettestsvc.service failed. See 'journalctl -xe' for details.
# journalctl -f
Dec 29 18:11:37 endeavour mount[12176]: mount.nfs: access denied by server while mounting nas:/mnt/nas/xtest
Dec 29 18:11:37 endeavour systemd[1]: var-tmp-scratch.mount mount process exited, code=exited status=32
Dec 29 18:11:37 endeavour systemd[1]: Failed to mount /var/tmp/scratch.
Dec 29 18:11:37 endeavour systemd[1]: Dependency failed for Test systemd service puppettestsvc.
Dec 29 18:11:37 endeavour systemd[1]: Job puppettestsvc.service/start failed with result 'dependency'.
Dec 29 18:11:37 endeavour systemd[1]: Unit var-tmp-scratch.mount entered failed state.

• Reboot testing is hard, because I’m using /run for the service definition. (/lib/systemd/system is for packages, /etc/systemd/system is for ‘site.’). The only way to simulate amending packaged unit configuration is via /run, and that gets wiped at reboot.  But, I’m happy from the last test that this works as expected.