Two main things covered off in this post:

• Getting libvirt-guests to work correctly, and
• machined/machinectl

libvirt-guests

This is configured via /etc/sysconfig/libvirt-guests and there’s a systemd service to enable: libvirt-guests.service

Enabling it allows you to configure how guests start and stop with more flexibility.

Some features libvirt-guests provides:

• On host shutdown, suspend the guests or shut down the guests (ON_SHUTDOWN)
• Specify how many seconds to give guests to shut down (SHUTDOWN_TIMEOUT)
• Wait a number of seconds between starting guests (or start them in parallel) – START_DELAY

The mechanism to space out guest startups might solve a problem where my puppet master daemon times out before starting – not enough CPU on my hosts to power all the processes in all the guests at boot time.

Having enabled it, it doesn’t look like it’s working correctly at startup.

kvm: 2 guests now active
libvirt-guests.sh: Resuming guests on default URI...
libvirt-guests.sh: Resuming guest test1: already active
libvirt-guests.sh: Resuming guest test2: already active
systemd: Started Suspend/Resume Running libvirt Guests.

Shutdown is fine.

libvirt-guests.sh[5590]: Running guests on default URI: test1, test2
libvirt-guests.sh[5590]: Shutting down guests on default URI...
libvirt-guests.sh[5590]: Starting shutdown on guest: test1
libvirt-guests.sh[5590]: Waiting for guest test1 to shut down, 300 seconds left
systemd[1]: Stopped Virtual Machine qemu-1-test1.
systemd-machined[3631]: Machine qemu-1-test1 terminated.
kvm[5727]: 1 guest now active
libvirt-guests.sh[5590]: Shutdown of guest test1 complete.
libvirt-guests.sh[5590]: Starting shutdown on guest: test2
libvirt-guests.sh[5590]: Waiting for guest test2 to shut down, 300 seconds left
kvm[5777]: 0 guests now active
systemd[1]: Stopped Virtual Machine qemu-2-test2.
systemd-machined[3631]: Machine qemu-2-test2 terminated.
libvirt-guests.sh[5590]: Shutdown of guest test2 complete.
systemd[1]: Stopped Suspend/Resume Running libvirt Guests.
systemd[1]: Stopping Virtualization daemon...
systemd[1]: Stopped target Libvirt guests shutdown.
systemd[1]: Stopped Virtualization daemon.

libvirt-guests is a shell script.

• The start subroutine loops over $LISTFILE and starts the specified virtual machines. This expands to /var/lib/libvirt/libvirt-guests.
  You won’t find this present when the system is booted because the start subroutine removes it when it’s done.
• The stop subroutine creates $LISTFILE.

libvirt-guests maintains its own state and configuration based on what was running when the system was shut down.

It doesn’t look at the libvirt autostart configuration, ie:

# virsh list --autostart
 Id    Name                           State
----------------------------------------------------
 1     test1                          running
 2     test2                          running

Libvirt is starting first, as it has to, starting the guests as configured; libvirt-guests loops over LISTFILE and finds them ‘already active.’

fix

The fix, non-intuitively, is to set the guests not to autostart in libvirt.

# virsh autostart --disable test1
Domain test1 unmarked as autostarted
# virsh autostart --disable test2
Domain test2 unmarked as autostarted

There’s some rough edges on libvirt-guests.

$LISTFILE becomes the autostart source of truth.

• Shut down a guest manually, and it then isn’t in LISTFILE, and stops starting at boot.
  Fix: start it manually, it’ll then make it into LISTFILE next shutdown.
• If libvirt-guests service fails at startup, systemd won’t call it at shutdown.
  • The guests are shut down by systemd as it sees fit.
  • No $LISTFILE is created, though the old one might still be there, so either nothing gets started, or the guests that get started might not be what was expected.
  • One way to trigger a failure is to get impatient waiting for the startup timeout you’ve configured. libvirt-guests handles this in a manner that systemd interprets as a failed startup.

The systemd configuration for libvirt-guests has comments to the effect that it’s been converted from init.d to systemd with limited changes. There maybe a number of ways in which systemd’s uncompromising philosophy means that libvirt-guests doesn’t always work as desired.

conclusion

There’s two ways ‘out of the box’ to run/start KVM guests on standalone hosts.

• using libvirt. Use this if you care about guests coming up at boot. One assumes therefore that you also have enough CPU to schedule all the guests.
• libvirt-guests.  This can help you squeeze more into the lab at home by managing startup and shutdown more flexibly, but note the rough edges above.

troubleshooting kvm at boot – persistent journald

Make journald persistent. Create /var/log/journal/ (easy!) or reconfigure journald from ‘auto’ to ‘persistent’.

If you are going to create /var/log/journal/ with puppet, it looks like journald has an opinion about the permissions. I’d configured puppet to enforce root:root 0755:

puppet-agent[6329]: (/Stage[main]/Profile::Journald/File[/var/log/journal]/group) 
                    group changed 'systemd-journal' to 'root'
puppet-agent[6329]: (/Stage[main]/Profile::Journald/File[/var/log/journal]/mode) 
                    mode changed '2755' to '0755'

Instead, enforce it as:

  file { '/var/lib/journal':
    ensure => directory,
    owner  => root,
    group  => systemd-journal,
    mode   => '2755',
  }

machined/machinectl

This is mainly a summary of what I found out from the man pages.

Man pages:

• systemd-machined.service(8)
• machinectl(1)

systemd-machined is a system service that keeps track of virtual machines and
containers, and processes belonging to them.

# systemctl status systemd-machined.service
● systemd-machined.service - Virtual Machine and Container Registration Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-machined.service; static; vendor preset: disabled)
   Active: active (running) since Fri 2019-04-19 16:27:23 BST; 49min ago
     Docs: man:systemd-machined.service(8)
           http://www.freedesktop.org/wiki/Software/systemd/machined
 Main PID: 3635 (systemd-machine)
   Status: "Processing requests..."
    Tasks: 1
   CGroup: /system.slice/systemd-machined.service
           └─3635 /usr/lib/systemd/systemd-machined

Apr 19 16:27:23 systemd[1]: Starting Virtual Machine and Container Registration Service...
Apr 19 16:27:23 systemd[1]: Started Virtual Machine and Container Registration Service.
Apr 19 16:27:30 systemd-machined[3635]: New machine qemu-1-test1.
Apr 19 16:27:34 systemd-machined[3635]: New machine qemu-2-test2.

# machinectl
MACHINE      CLASS SERVICE     
qemu-1-test1 vm    libvirt-qemu
qemu-2-test2 vm    libvirt-qemu

2 machines listed.
# machinectl status qemu-1-test1
qemu-1-test1(368cc74328a74bd8bebbe70135f4455e)
           Since: Fri 2019-04-19 16:27:30 BST; 50min ago
          Leader: 4660 (qemu-kvm)
         Service: libvirt-qemu; class vm
           Iface: vnet0
            Unit: machine-qemu\x2d1\x2dtest1.scope
                  └─4660 /usr/libexec/qemu-kvm -name test1 -S -machine 
                         pc-i440fx-rhel7.0.0,accel=kvm,usb=off,
                         dump-guest-core=off -m 1024 -re

Apr 19 16:27:30 systemd[1]: Started Virtual Machine qemu-1-test1.

From the machinectl man page:

       -o, --output=
           When used with status, controls the formatting of the journal entries 
           that are shown. For the available choices, see journalctl(1).
           Defaults to "short".
       -H, --host= Execute the operation remotely. 

So, in part, machinectl is a way of interacting with systemctl and journald.  One wonders how remotely would work.

It has a download mechanism.  The man page has an example of downloading a raw disk file and starting it as a virtual machine.

There maybe an alternative way to deal with a stubborn guest that won’t die:

       terminate NAME...
           Immediately terminates a virtual machine or container, without 
           cleanly shutting it down.

It’s also got lots of references to containers, functionality that one might expect to use, say, docker tools to achieve.

       start NAME...
           Start a container as a system service, using systemd-nspawn(1)
           [..]
           Starting VMs and container images on (a variety of other container 
           and VM managers) requires manager-specific tools.

       login NAME
           Open an interactive terminal login session to a container.

       enable NAME..., disable NAME...
           Enable or disable a container as a system service to start at 
           system boot

       copy-to NAME PATH [PATH]
           Copies files or directories from the host system into a running 
           container. 

That’s a selection of the container commands.

Unclear from the man page if ‘start’ would work with KVM guests, but they have an example of firing one up off a raw disk and connecting into it.

It has VM specific options like list-images, but those don’t seem to be aimed at libvirt/KVM, as it doesn’t locate the disk files in /var/lib/libvirt/images. This suggests that it provides a non-libvirt way to run guests.

       Machine images are preferably stored in /var/lib/machines/, 
       but are also searched for in /usr/local/lib/machines/ and 
       /usr/lib/machines/. For compatibility reasons the directory 
       /var/lib/container/ is searched, too.