I’m moving away from using a Puppet server, and having pushed the first few commits for this change, I realised I needed to deploy the changes on a client that was managed off the server.

Turns out I built my Puppet server just over five years ago:

Info: Not using expired certificate for ca from cache; expired at 2021-11-19 21:06:12 UTC

Next problem: Puppet 2016.4 docs have been taken offline.  The procedure for regenerating a monolithic master’s CA is now markdown in a git repo.  For a monolithic install, rebuilding the Puppet CA from scratch is probably the easiest approach.

I tried it three times, and could not get any Puppet Agent to do the TLS handshake, including the one on the Puppet Master – which was really odd.  I’d nuked all the client and server TLS, so why couldn’t it handshake with itself?

Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect SYSCALL returned=5 errno=0 state=unknown state
Info: Retrieving pluginfacts
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect SYSCALL returned=5 errno=0 state=unknown state
Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect SYSCALL returned=5 errno=0 state=unknown state

TL;DR: Turn it off and turn it on again.

This error is a TLS trust issue, it’s just that the Puppet Server terminates the connection really quickly and so the client doesn’t generate a useful error.

I ended up shutting down the VM in frustration, briefly intending never to start it up again.  However, I powered it back on for one last try, and behold .. it was working.  Something must have had an old CA in memory, and isn’t restarted by the procedure for rebuilding the CA:

puppet enterprise configure --no-recover

Now the server’s working, you:

rm -rf /etc/puppetlabs/puppet/ssl
puppet agent --waitforcert 20 --test --server=puppet.$(hostname -d)    --certname="$(hostname -f)" --environment=foo
  # server side
puppet cert list
puppet cert sign client.domain