Not to be confused with TLS (was: SSL) certificates.

why?

Problem 1: Does anyone check SSH server keys before accepting them?

The authenticity of host 'server (192.168.1.13)' can't be established.
ECDSA key fingerprint is SHA256:P45XQklRehB7js1bfqpbRX+dq0vTEQoJbcwACNrSGow.
ECDSA key fingerprint is MD5:d2:4a:3d:a1:59:ec:30:fe:36:11:1a:61:7c:f1:3f:eb.
Are you sure you want to continue connecting (yes/no)

A mechanism to distribute the known keys is needed. IPA/Redhat IDM does this, but it only works for IPA clients. For clients that’ll never be in that ecosystem, like Windows, not so much. (Whether SSH certificates solves for Windows, Putty etc., I don’t know ..)

Problem 2: Clustered SSH solutions.

Using a couple of servers load balanced to provide an SSH service, such as bastion hosts or for git clients, would generally require them to have the same host keys.

overview

It’s handled by ssh-keygen.  There’s a CERTIFICATES section in the man page:

ssh-keygen supports signing of keys to produce certificates that may be used for user or host authentication. Certificates consist of a public key, some identity information, zero or more principal (user or host) names and a set of options that are signed by a Certification Authority (CA) key. Clients or servers may then trust only the CA key and verify its signature on a certificate rather than trusting many user/host keys.

Other points from the man page:

• By default, generated certificates are valid for all users or hosts.
• By default, certificates are valid from UNIX Epoch to the distant future. Certificates may be defined with a validity lifetime.
• For certificates to be used for user or host authentication, the CA public key must be trusted by sshd(8) or ssh(1).
• Limitations on the validity and use of user certificates may be specified through certificate options. At present, no options are valid for host keys.

I’m only going to look at certificates for hosts.

Features I’m not going to consider:

• Validity interval (-V) – for the reasons that short lived certs are good for TLS, it’s probably the same for SSH.  However, the cert is a proxy for the host key. I guess the vast majority of server SSH keys are created at first boot, and are expired only when the server is decommissioned.
• Serial numbers (-z) – this caught my eye because I know TLS serial numbers are randomized as a workaround for potential attacks.  No basis for thinking it’s a problem with SSH.
• Key revocation lists. This worked out so well for TLS! If you plan to revoke keys, you need to find a way to distribute the revoked keys.  The main issue with SSH keys is how to distribute them. Solve the problem for revoked certificates, and you don’t need to use certificates any more. Just distribute valid host keys.
• User certificates.

TL;DR

• Consider what principals you need for the host certificates: Hostname with domain, hostname without domain, IP addresses?
• Create one or more CAs and as a minimum armour the private keys with pass phrases.
• Consider which flavours of SSH hostkey you want to use, eg: just ed25519. If you have, say, RSA enabled, then either sign those hostkeys was well, or they won’t be trusted.
• Serve up certificate(s) on each server using HostCertificate in sshd_config
• Trust host certificates across your estate using ‘@cert-authority wildcard …’ entries in /etc/ssh/known_hosts

the certificate authority

• Generate the CA key pair. A pass phrase provides a basic level of protection for the CA private key, and is probably a Good Idea (TM).

mkdir -p test-ssh-ca/certs test-ssh-ca/ca
  # to ease copying over the public keys
chmod 1777 test-ssh-ca/certs
  # to keep 
chmod 700 test-ssh-ca/certs
ssh-keygen -t ed25519 -f test-ssh-ca/ca/ca_ed25519

• the server SSH public key is needed on the CA to produce a certificate
• a certificate identity or key_id is required (-I), see details below
• the principals (-n) I guess determine the validity of the cert, like TLS SANs
• the certificate file is the same as the key filename, with -cert appended

# copy over the certs
for i in ed25519 rsa ; do scp /etc/ssh/ssh_host_${i}_key.pub \
user@cahost:/path/test-ssh-ca/certs/ssh_$(hostname)_${i}_key.pub ; done

  # create and query certificate
h="foo"; d="domain"; ds=$(date '+%Y%m%d%H%M')
for i in ed25519 rsa ; do
  ssh-keygen -s ca/ca_ed25519 -n ${h}.${d} -I ${h}.${d}-${ds} -h \
        certs/ssh_${h}.${d}_${i}_key.pub
  ssh-keygen -L -f certs/ssh_${h}.${d}_${i}_key-cert.pub
done

I signed both the RSA and Edwards-curve keys with the Ed certificate authority. I get the impression that RSA has a limited future (all DSA flavours are even more limited) so it’d be simpler to trust only a ed25519 CA. The public keys are shorter as well.

openssh doesn’t seem to mind:

Type: ssh-rsa-cert-v01@openssh.com host certificate
Public key: RSA-CERT SHA256:evfmX77lje25PsCxq177bnLsHG7qV1QZ/CvNLydrcOo
Signing CA: ED25519 SHA256:5tEb2MngpmsQYaFtKtD/+5e1gi2M3qpfk1RWr72v0C4
Key ID: "foo.bar-202101051400"
Serial: 0
Valid: forever
Principals: 
        foo.bar
Critical Options: (none)
Extensions: (none)

As indicated in the TL;DR section, deciding whether you need RSA at all is probably the best approach.  As indicated by my testing, servers can have a certificate for each flavour of public key, which one is used seems to be determined by the client.

certificate_identity / key_id

Must specify key id (-I) when certifying

It seems, from stack exchange, that it can be set to anything. From ssh-keygen man page:

-I certificate_identity
        Specify the key identity when signing a public key.

In all cases, key_id is a "key identifier" that is logged by the 
server when the certificate is used for authentication.

versions

Server and client are Centos 7: 7.4p1-21.el7
Being sourced from RHEL, it’s possible that it behaves differently from stock openssh 7.4 if fixes have been backported.

ssh server configuration

From the man pages:

# ssh-keygen
For certificates to be used for user or host authentication, the CA public key 
must be trusted by sshd(8) or ssh(1).

# sshd
-c host_certificate_file
   Specifies a path to a certificate file to identify sshd during key exchange.
   The certificate file must match a host key file specified using the -h option
   or the HostKey configuration directive.

# sshd_config
HostCertificate
  Specifies a file containing a public host certificate. The certificate's public
  key must match a  private host key already specified by HostKey. The default 
  behaviour of sshd(8) is not to load any certificates.

Questions I have so far:

• The default behaviour of sshd is pretty important.  I can’t find anything else in either man page.  Does specifying a HostCertificate change this, and if so, why mention it? sshd’s ‘-c host_certificate_file’ param seems to do the same thing, but default behaviour isn’t mentioned there.
• Can more than one certificate be served up.  There’s more than one host key, after all.

testing

Test 1: ed25519, host key known

• added HostCertificate entry to /etc/ssh/sshd_config, specifying the ed25519 certificate
• restarted sshd via systemd
• client ~/.ssh/known_hosts contained the hostkey
• initiated client connection (with -vvv debugging)

Findings:

• Nothing in debug output about the certificate, known host key did all the heavy lifting.

Test 2: ed25519, host key not known

• Remarked out known_hosts entry

Findings:

• debug output is different.

debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com
        SHA256:rv8W+p2FNJjwTRw/UgyHrrZhSY/flBI8vgIkNdwX/Zc, 
        serial 0 ID "foo.bar-202101051400" CA ssh-ed25519
        SHA256:5tEb2MngpmsQYaFtKtD/+5e1gi2M3qpfk1RWr72v0C4 valid forever
debug2: Server host certificate hostname: foo.bar
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: No matching CA found. Retry with plain key

Tests 3,4: rsa and ed25519 certificates, host key not known

• added second HostCertificate entry to /etc/ssh/sshd_config, specifying the rsa certificate
• restarted sshd via systemd
• the tests differed by sequence: RSA second or first.

Findings:

• Same as test 2: ssh-ed25519-cert-v01 is served up.
• Ordering of HostCertificate entries made no difference.

Test 5: rsa and ed25519 certificates, host key not known, try to force RSA.

Goal is to verify if the server recognizes both certificates. Based on the difference between the known and unknown host key tests, the client seems to drive what’s served up.  Had to track down HostKeyAlgorithms in the ssh_config man page.

  # client parameters #1
ssh -vvv -o HostKeyAlgorithms=ssh-rsa
  # client parameters #2
ssh -vvv -o HostKeyAlgorithms=ssh-rsa-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa

Findings:

• Parameters #1: no certificate is requested
• Parameters #2: RSA cert served up, RSA key as backup.

debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com 
        SHA256:evfmX77lje25PsCxq177bnLsHG7qV1QZ/CvNLydrcOo,
        serial 0 ID "foo.bar-202101051400" CA ssh-ed25519
        SHA256:5tEb2MngpmsQYaFtKtD/+5e1gi2M3qpfk1RWr72v0C4 valid forever
debug2: Server host certificate hostname: foo.bar
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: No matching CA found. Retry with plain key
The authenticity of host 'foo (192.168.1.13)' can't be established.
RSA key fingerprint is SHA256:evfmX77lje25PsCxq177bnLsHG7qV1QZ/CvNLydrcOo.

conclusions

• Providing HostCertificate entries in sshd_config enables the feature
• Multiple certs can be provided for key pairs of different algorithms.
• The key pair algorithm determines which certificate is requested by the client.  Both my test certs came from a ed25519 CA, but the certificate for the RSA key was selected when the client was set to favour RSA.
• Only one certificate will be requested by the client.

Signing RSA public keys with ed25519 might prove troublesome with some clients, probably old ones I suppose.  I think I’d be minded to scrap RSA hostkeys for my servers anyway, as the clients are defaulting to ed25519 anyway, and then I don’t need certificates for the RSA keys.

ssh services

If you have a number of servers backing an SSH service like bastion hosts or git, you could use server certificates.

• Each backend server added to the service has a certificate that specifies all the principals that server is providing: it’s hostname, the SSH service address (‘dmz.company.net’, ‘git.company.net’ etc.)
• The server keys are all different, but they can all provide the service because of the specified principal.
• Spoilers: the clients then either trust the certificate for a broad range of hostnames, as part of a broader use of certificates for SSH, or just for that specific service address: ie: the known_hosts file specifies just the shared principal in the second field.

ssh client setup

Oddly, the mechanism for trusting a certificate seems to be in the sshd (server) man page.

# ssh-keygen
For certificates to be used for user or host authentication, the CA public key  
must be trusted by sshd(8) or ssh(1).

# sshd
SSH_KNOWN_HOSTS FILE FORMAT
  The /etc/ssh/ssh_known_hosts and ~/.ssh/known_hosts files contain host public
  keys for all known hosts.
  [..]
  Each line in these files contains the following fields: markers (optional)
  [..]
  The marker is optional, but if it is present then it must be one 
  of “@cert-authority”, to indicate that the line contains a certification
  authority (CA) key,

It wasn’t clear to me how to configure the fields so I googled it. Hat tip: James Fisher.

An entry might look like this:

@cert-authority *.bar ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrhNtIjeTAFhyqQYu188TXXIMXFAHzi+ErXM/jwm1FS testca

It look a bit of testing to get it to work.

• The ssh client effectively checks the hostname validity twice. First, it checks the second field of any CA key entries (here: ‘*.bar’) and it checks that it matches the servername you’re connecting to.  In this case, servers in domain ‘bar’.

debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com
        SHA256:rv8W+p2FNJjwTRw/UgyHrrZhSY/flBI8vgIkNdwX/Zc,
        serial 0 ID "foo.bar-202101051400" CA ssh-ed25519 
        SHA256:5tEb2MngpmsQYaFtKtD/+5e1gi2M3qpfk1RWr72v0C4 valid forever
debug2: Server host certificate hostname: foo.bar
debug3: hostkeys_foreach: reading file "/home/ben/.ssh/known_hosts"
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: No matching CA found. Retry with plain key
The authenticity of host 'foo (192.168.1.13)' can't be established.
ED25519 key fingerprint is SHA256:rv8W+p2FNJjwTRw/UgyHrrZhSY/flBI8vgIkNdwX/Zc.

• Note: ‘No matching CA found’. Asterisk will assure a match.
• The second hostname check is that whatever’s specified in the SSH client parameters must match one of the certificate principals.  Want to ssh to your servers without the domain? You need to add the unqualified hostnames as principals explicitly.

$ ssh -vvv foo
<snip>
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com
        SHA256:rv8W+p2FNJjwTRw/UgyHrrZhSY/flBI8vgIkNdwX/Zc,
        serial 0 ID "foo.bar-202101051400" CA ssh-ed25519 
        SHA256:5tEb2MngpmsQYaFtKtD/+5e1gi2M3qpfk1RWr72v0C4 valid forever
debug2: Server host certificate hostname: foo.bar
debug3: hostkeys_foreach: reading file "/home/user/.ssh/known_hosts"
debug3: record_hostkey: found ca key type ED25519 in file /home/user/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from foo
debug3: hostkeys_foreach: reading file "/etc/ssh/ssh_known_hosts"
debug1: Host 'foo' is known and matches the ED25519-CERT host certificate.
debug1: Found CA key in /home/user/.ssh/known_hosts:2
key_cert_check_authority: invalid certificate
Certificate invalid: name is not a listed principal
debug1: No matching CA found. Retry with plain key
The authenticity of host 'foo (192.168.1.13)' can't be established.
ED25519 key fingerprint is SHA256:rv8W+p2FNJjwTRw/UgyHrrZhSY/flBI8vgIkNdwX/Zc.

• Ditto IP address. You’ll need that as a principal if you want host key validation to work via the certificate.  Providing ..
• IP addresses will only be checked if the second field will scope that CA to IP addresses. Asterisk will do that, so will:

@cert-authority 192.168.*.*,*.bar ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrhNtIjeTAFhyqQYu188TXXIMXFAHzi+ErXM/jwm1FS testca

The validity mechanism in known_hosts could allow for multiple CAs based on server function or anything else that could be defined in that second field. For example, different CAs for different subsidiaries or different clients’ servers, or for different server functions.

user certificates

I’m not currently thinking about using these, but I found a guide to doing it on James’s Fisher’s website, I noticed in the man pages:

# sshd

AUTHORIZED_KEYS FILE FORMAT
  principals="principals"
    On a cert-authority line, specifies allowed principals for certificate  
    authentication as a comma-separated list. At least one name from the list  
    must appear in the certificate's list of principals for the certificate  
    to be accepted. This option is ignored for keys that are not marked as 
    trusted certificate signers using the cert-authority option.

# sshd_confug

AuthorizedPrincipalsFile
  Specifies a file that lists principal names that are accepted for
  certificate authentication.
  <snip>

TrustedUserCAKeys
  Specifies a file containing public keys of certificate authorities that 
  are trusted to sign user certificates for authentication, or none to 
  not use one. Keys are listed one per line; empty lines and comments 
  starting with ‘#’ are allowed. If a certificate is presented for authentication
  and has its signing CA key listed in this file, then it may be used for 
  authentication for any user listed in the certificate's principals list. 
  Note that certificates that lack a list of principals will not be permitted 
  for authentication using TrustedUserCAKeys.

# ssh

-i identity_file
  <snip>
  If no certificates have been explicitly specified by the CertificateFile 
  directive, ssh will also try to load certificate information from the 
  filename obtained by appending -cert.pub to identity filenames.

-o option
  CertificateFile

-Q query_option
  Queries ssh for the algorithms supported for the specified version 2.
  <snip>
  key-cert (certificate key types), key-plain (non-certificate key types),

A variation on public key authentication is available in the form of 
certificate authentication: instead of a set of public/private keys,
signed certificates are used.

The most convenient way to use public key or certificate authentication 
may be with an authentication agent. See ssh-agent(1) and (optionally) 
the AddKeysToAgent directive in ssh_config(5) for more information.