LeoMoon Wiki-Go calls itself a modern, feature-rich, databaseless, flat-file wiki platform built with Go, and it is all that.
Wiki-Go is a single-binary Go program, but there is a docker image for those who want it. I download the binary for my platform and launch it; this creates a data/ directory and its configuration file. If need be I configure a port number for it plus a timezone and the odd other setting and relaunch the binary.
The program has all the features I need and more, and what I most appreciate is that I can simply drop Markdown files, images, videos, etc. into a directory to have Wiki-Go serve them. I can also move documents to other pages, so for instance, when I’ve finished working on the Cascade page, I’ll move it into DNS/.
The program also boasts an interactive Markdown editor which certainly fulfills all I expect of one, including being able to format text, add links, insert images, etc.
In addition to wiki pages, the program supports the creation of Kanban boards and Link management pages, all of which are stored in *.md Markdown files.
There is a REST API with examples, which might be interesting to programmatically add comments, say, but if I have access to the server, I copy files into their directories.
There’s a web-based settings dialog where I can add users and configure roles, but I can also add these to the YAML configuration file manually:
This is really a lovely piece of work, and it is open source, and IMO its major features: Markdown in flat files on the file system, which makes it easy for me to synchronize the data directory onto my laptop and use a separate instance of Wiki-Go on that.
Assuming we want a host’s private key to be generated on a node and reside on the node only, and assuming our SSH CA (certification authority) is on the Ansible controller, we can use delegation to localhost for the bits which should happen on the controller.
In the following simple playbook I use a block in which I create a temporary unique directory locally into which the public key from the host is copied and signed, and the always portion of the block ensures the directory is cleared out even on error within the block. The public portion of the SSH key pair is returned in openssh_keypair’s metadata.
-hosts:d13gather_facts:yesremote_user:jpvars:dirname:"/etc/ssh"tasks:-name:Generate ED25519 SSH host key on nodecommunity.crypto.openssh_keypair:backend:"opensshbin"path:"{{dirname}}/ssh_host_ed25519_key"comment:"ansible-made™forhost{{inventory_hostname}}"type:"ed25519"become:trueregister:keydata-debug:var=keydata-block:-name:Create a local temporary directoryansible.builtin.tempfile:prefix:"jp"suffix:"cert"state:directoryregister:pdelegate_to:localhost-name:Save public host key to local temporary fileansible.builtin.copy:content:"{{keydata.public_key}}"dest:"{{p.path}}/{{keydata.filename|basename}}"delegate_to:localhost-name:Sign SSH certificate on local copy of public host keycommunity.crypto.openssh_cert:identifier:"{{inventory_hostname}}"public_key:"{{p.path}}/{{keydata.filename|basename}}"principals:"{{[ansible_fqdn]+ansible_all_ipv4_addresses}}"path:"{{p.path}}/{{keydata.filename|basename}}-cert.pub"serial_number:10signing_key:"../CA/ssh-ca"use_agent:truetype:"host"valid_from:"+0m"valid_to:"+53w"delegate_to:localhost-name:Install signed certificate on target nodeansible.builtin.copy:src:"{{p.path}}/{{keydata.filename|basename}}-cert.pub"dest:"{{dirname}}"mode:0444become:truealways:-name:Remove local temporary directoryansible.builtin.file:path:"{{p.path}}"state:absentdelegate_to:localhost
The openssh_cert module is able to sign the public key and create a certificate without specifying a password because I have the CA key in my SSH agent.
I left the debug in there so we can see the returned metadata when the host key is generated or read if it need not be changed:
TASK [Generate ED25519 SSH host key on node] *************************************************
changed: [d13]
TASK [debug] *********************************************************************************
ok: [d13] => {
"keydata": {
"changed": true,
"comment": "ansible-made™ for host d13",
"failed": false,
"filename": "/etc/ssh/ssh_host_ed25519_key",
"fingerprint": "SHA256:3zm2UIyqO0xsxejndKdg9+Rpm4JeGuxamlsZwNOGX2Y",
"public_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBMNncuSW1uwBwCQTb7ZOH61ChYQiJVYtGR/SyuD3Mcl",
"size": 256,
"type": "ed25519"
}
}
TASK [Create a local temporary directory] ****************************************************
changed: [d13 -> localhost]
TASK [Save public host key to local temporary file] ******************************************
changed: [d13 -> localhost]
TASK [Sign SSH certificate on local copy of public host key] *********************************
changed: [d13 -> localhost]
TASK [Install signed certificate on target node] *********************************************
changed: [d13]
TASK [Remove local temporary directory] ******************************************************
changed: [d13 -> localhost]
Getting the playbook to alter sshd’s configuration to include the new certificate and restart it on a handler or two ought to be simple enough.
When I ssh into a server for the first time, I’m confronted with a dialog which asks me to verify I’m actually talking to the machine I expect to be talking to.
$ssh -l jp 192.0.2.65
The authenticity of host '192.0.2.65 (192.0.2.65)' can't be established.
ED25519 key fingerprint is SHA256:4WTRnq2OR1m03TpnHCfkFdlh1gN/PBXE4vDi0WnjFEc.
No matching host key fingerprint found in DNS.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?
It is likely that the majority of users cross their fingers and type ‘yes’, which is not really a clever response. This Trust on First Use (TOFU) is what permits SSH to ensure that my SSH client verifies which server it’s talking to. I ought to have asked the administrator of the server to tell me its fingerprint, and if I am the administrator I ought to know how to do this: (in the following examples, a shell prompt % indicates I’m working as `root)
If the two fingerprints compare equal, I can trust that I am connecting to the correct server and can continue with ‘yes’ or I paste a known host fingerprint into the prompt: trust on first use is accomplished. (Utilities such as ssh-keyscan gather public keys from remote hosts, but I still ought to verify out-of-band whether I’m talking to the correct machine. SSH fingerprints can also be in the DNS, what can’t?, but that’s a different story.)
The session then possibly continues with me being asked for the target user’s password which, if entered correctly, grants me access to the machine.
SSH key pairs
If I create an SSH key pair, install my public key in the correct location (typically $HOME/.ssh/authorized_keys on the target node), and present the private key upon connection, then I don’t need to type the target user’s password; instead I enter the key’s passphrase, a hopefully much more complicated combination of words, to unlock the private key. I say “hopefully”, because this passphrase is what encrypts the private key so that it cannot be used without it. (Note that I typically generate keys with a comment in them so as to more easily keep track of them (-C), and specify which file (-f) they should be written to. The comment can be read from the public key file and will later be visible in the SSH agent.)
$ssh-keygen -t ecdsa -C"JP's demo key"-f demokey
Generating public/private ecdsa key pair.
Enter passphrase for "demokey" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in demokey
Your public key has been saved in demokey.pub
$ssh-copy-id -i demokey.pub jp@192.0.2.65
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "demokey.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
jp@192.0.2.65's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh -i ./demokey 'jp@192.0.2.65'"
and check to make sure that only the key(s) you wanted were added.
$ssh -i demokey jp@192.0.2.65
Enter passphrase for key 'demokey':
jp@node:~$
The private SSH key still needs unlocking on use because I set a passphrase on it when creating it.
I can avoid having to do that at every use of the key, by launching an SSH agent which I feed with my private key and it will then no longer requests a passphrase on use. Desktop systems typically launch an agent at startup, but I do so explicitly here to demonstrate:
In order for this to work, the server needs a copy of the public key I use in one of the locations specified by SSHd’s AuthorizedKeysFile configuration which defaults to .ssh/authorized_keys and .ssh/authorized_keys2. Public keys can also be sourced from a command on a node.
This is well known and has worked for very many years, but the required procedures for public key authentication to work come with some disadvantages:
a copy of my SSH public key has to be available for each user I want to login as on a node
TOFU typically causes the host fingerprint to be stored on my client (in the known_hosts file)
when a host key rolls, I get a big warning
What’s with this warning? Well, if the server’s SSH host key changes, for instance because the server has been re-installed without restoring its original host keys, or because an administrator has forcefully re-generated them, my client will loudly complain that the server’s host keys have changed. Let’s see this in action.
The server’s admin re-generates SSH host keys on the server and restarts sshd
% rm -f /etc/ssh/ssh_host_*
% ssh-keygen -A
ssh-keygen: generating new host keys: RSA ECDSA ED25519
% systemctl restart sshd
Now I try and access the same server again, as above (my user key is still in the agent). The connection fails, I then remove the offending host key from the known_hosts file using ssh-keygen, obtain the server’s host key fingerprint to re-validate TOFU, and finally accept the host key into my known_hosts again.
$ssh -l jp 192.0.2.65 df-h /
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:3zm2UIyqO0xsxejndKdg9+Rpm4JeGuxamlsZwNOGX2Y.
Please contact your system administrator.
Add correct host key in /Users/jpm/.ssh/known_hosts to get rid of this message.
Offending ED25519 key in /Users/jpm/.ssh/known_hosts:649
Host key for 192.0.2.65 has changed and you have requested strict checking.
Host key verification failed.
$ssh-keygen -R 192.0.2.65
#Host 192.0.2.65 found: line 649
/Users/jpm/.ssh/known_hosts updated.
Original contents retained as /Users/jpm/.ssh/known_hosts.old
$ssh -l jp 192.0.2.65 df-h /
The authenticity of host '192.0.2.65 (192.0.2.65)' can't be established.
ED25519 key fingerprint is SHA256:3zm2UIyqO0xsxejndKdg9+Rpm4JeGuxamlsZwNOGX2Y.
No matching host key fingerprint found in DNS.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.0.2.65' (ED25519) to the list of known hosts.
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 1.2G 17G 7% /
Admittedly this probably won’t occur very frequently, but in larger environments, this is something our users need to be made aware of, including how to correctly remedy the situation.
For those of us with a small handful of nodes or requiring connections to systems we have no root privileges on, working with keys as described so far is likely sufficient respectively the only choice, however for those with dozens or even hundreds of servers and full control there-over, we can make all of the disadvantages above go away with an SSH CA (certification authority) and SSH certificates. That sounds complicated, but it isn’t.
SSH Certification Authority
Those familiar with X.509 certificates, their complexity, and certification authorities might well have begun to groan now, but rest assured an SSH CA is something quite simple: all we need is an SSH key pair, and a few additional options for the ssh-keygen utility we’re already familiar with!
SSH certificates have existed in OpenSSH since version 5.4 released in March 2010, and the certificate format is based on data formats which implementations already support – one reason OpenSSH designed it this way rather than using the far more complicated X.509 format. The following looks like an SSH public key file is actually an OpenSSH public certificate.
For the following experiments I will use a single SSH CA key pair for users and hosts. (Two key pairs, one of hosts another for users, are sometimes used for better separation, but the principles described in the following are applied in the same way.)
Let me begin by enumerating some of the advantages of using SSH certificates:
no more deploying of public SSH keys into server authorized_keys files; users can forget about ssh-copy-id, editing $HOME/.ssh/authorized_keys manually, etc.
no danger of a user adding a key to an authorized_keys file for a key which shouldn’t have access at all
a server’s host keys can be rolled (replaced) without the scary “WARNING” on clients as we saw earlier; in fact there will no longer be host keys added to users’ known_hosts files!
no more TOFU (Trust on First Use) confirmation is required, as users will implicitly trust servers and vice-versa
principal names on user certificates dictate as which user(s) a user’s key can login as
remote commands can be enforced (user restrictions are bound to certificates and do not need to be added to a target user’s $HOME/.ssh/authorized_keys)
source IP prefixes can be limited, e.g. connecting as user root is permitted from 192.0.2.53/32 only.
certificates are valid for a specified time frame only, e.g. 30 minutes, 24 days, 34 weeks, etc. and they expire automatically (watch for clock drift)
a single line of configuration in the global known hosts files (e.g. /etc/ssh/ssh_known_hosts) suffices for all users on the client (but this line can also be in my personal $HOME/.ssh/known_hosts if I prefer)
in order to connect to a node, ssh -i needs the secret key file (jane) alongside its certificate (jane-cert.pub); if the certificate file isn’t available password authentication will be offered
So in our example the SSH CA will be used to sign host keys and user keys. To clarify, the CA is an SSH key pair which will be trusted by our nodes.
I proceed as follows.
On the CA machine, i.e. the system on which I will be “operating” the certification authority, I create a dedicated directory for the CA key pair. All we need on this system are the OpenSSH utilities, and we will be signing public keys on this system. I generate a CA key pair of algorithm ECDSA which generates short but strong keys; use any algorithm you prefer.
$umask 077;mkdir CA
$ssh-keygen -t ecdsa -C"JP's SSH CA"-f CA/ssh-ca
Generating public/private ecdsa key pair.
Enter passphrase for "CA/ssh-ca" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in CA/ssh-ca
Your public key has been saved in CA/ssh-ca.pub
The key fingerprint is:
SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM JP's SSH CA
$chmod-w CA/ssh-ca*
Jane creates a key pair for herself (or she uses an existing key pair she already has, e.g. in $HOME/.ssh/id_rsa or id_ecdsa). Like myself, she likes specifying a comment for the key and the file name into which it should be saved. Jane will send her public key (*.pub) to the CA for signing; as the public key is public it can be transmitted by any (also insecure) means.
$ssh-keygen -t ecdsa -C"Jane's key"-f jane
Generating public/private ecdsa key pair.
Enter passphrase for "jane" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in jane
Your public key has been saved in jane.pub
The key fingerprint is:
SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE Jane's key
Now I sign that user’s public key with our SSH CA key. I give the certificate an identity (-I), specify principals it may be used for (-n) (i.e. users on target hosts), a serial number (-z) for my own use and a validity (-V), in this example ending one week from now. I send the public certificate in jane-cert.pub back to Jane who then places the file adjacent to her private key file.
$ssh-keygen -s CA/ssh-ca -I"Jane Jolie"-n jane -z 001 -V +1w jane.pub
Enter passphrase for "CA/ssh-ca":
Signed user key jane-cert.pub: id "Jane Jolie" serial 1 for jane valid from 2026-03-27T13:57:00 to 2026-04-03T14:58:39
$ssh-keygen -L-f jane-cert.pub
jane-cert.pub:
Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
Public key: ECDSA-CERT SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE
Signing CA: ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM (using ecdsa-sha2-nistp256)
Key ID: "Jane Jolie"
Serial: 1
Valid: from 2026-03-27T13:57:00 to 2026-04-03T14:58:39
Principals:
jane
Critical Options: (none)
Extensions:
permit-X11-forwarding
permit-agent-forwarding
permit-port-forwarding
permit-pty
permit-user-rc
I then copy our CA’s public SSH key to the target server and configure it into the SSH server’s configuration. This ensures the SSH service on this node will trust SSH keys signed by this CA. I don’t restart sshd yet.
$[copy CA's public key to server]
% install -m444 /tmp/ssh-ca.pub /etc/ssh/ssh-ca.pub
% echo "TrustedUserCAKeys /etc/ssh/ssh-ca.pub" >> sshd_config
I then obtain the node’s host key(s) and sign them with our CA, again specifying a validity, an identifier of choice, a serial number, and a principal (the node’s hostname(s)) for the key. Note the -h option with which we sign a host certificate.
% [ copy server's /etc/ssh/ssh_host_ed25519_key.pub to CA ]
$ssh-keygen -h-s CA/ssh-ca -V +52w -I deadbeef01 -z 1000 -n alice.example.com ssh_host_ed25519_key.pub
Enter passphrase for "CA/ssh-ca":
Signed host key ssh_host_ed25519_key-cert.pub: id "deadbeef01" serial 1000 for alice.example.com valid from 2026-03-27T14:04:00 to 2027-03-26T14:05:47
$ssh-keygen -L-f ssh_host_ed25519_key-cert.pub
ssh_host_ed25519_key-cert.pub:
Type: ssh-ed25519-cert-v01@openssh.com host certificate
Public key: ED25519-CERT SHA256:ddxT1zL+HhHpIT5qWPdMJ6GC1SbVp2ij/2Sca5xA3RE
Signing CA: ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM (using ecdsa-sha2-nistp256)
Key ID: "deadbeef01"
Serial: 1000
Valid: from 2026-03-27T14:04:00 to 2027-03-26T14:05:47
Principals:
alice.example.com
Critical Options: (none)
Extensions: (none)
I install the signed certificate(s) alongside the SSH server’s host key(s)
$[ copy host's key certificate to the server ]
% install -m444 tmpfile /etc/ssh/ssh_host_ed25519_key-cert.pub
and configure the SSH server to use the new certificate and restart sshd
and finally add a reference to the host’s certificate authority in the client’s known_hosts file, either the global one in /etc/ssh/known_hosts or my own in ~/.ssh/known_hosts. Here I overwrite the file as I no longer need the fingerprints it contained, but I can also keep the file’s content and append or prepend this line. The @cert-authority line has a glob-style pattern used to match trusted nodes which present host keys signed with this (our CA’s) public key. This is how we add trust to our client for nodes’ SSH host keys signed by our CA.
We created an SSH CA by generating an SSH key pair, and we have signed one or more public user keys and public host keys. Then on a node, we wired up the public key of our SSH CA and made the server trust it.
On our client machines, we added a single line of configuration to our known_hosts file (user-specific or the global one), and this line will cause our clients to trust host keys signed by our CA.
We can now test.
An initial connection
In order to test, I create a user on the target system
% useradd -c "Jane Jolie" -m jane
we can now connect to the server. For debugging purposes I specify options (-o) to ensure ssh uses exactly the files I specify here. In particular I want to make sure we’re informed when server host keys cannot be validated with the ask option. If all goes well I will not be asked to trust a server’s fingerprint, and after the SSH session terminates, there will be no additional line in the UserKnownHostsFile. (This last file contains one line only, the @cert-authority line.)
$ssh -oUserKnownHostsFile=./known_hosts -oStrictHostKeyChecking=ask \-oIdentitiesOnly=true-i jane -l jane alice.example.com uname Linux
the server logs in auth.log (here OpenBSD) but likewise in a systemd journal. Note we can read out the fingerprint of the used key, its identity (Jane Jolie), its serial number (001 used during signing is simply an integer which is why it’s shown here as 1), and the public key fingerprint of the signing CA.
sshd-session[3099]: Accepted publickey for jane from 192.0.2.42 port 17087 ssh2: ECDSA-CERT SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE ID Jane Jolie (serial 1) CA ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM
If I want to connect by IP address as well as host name, I have to sign a node’s host key(s) with more than one principal name in them, and the entry in known_hosts needs these added comma-separated to it
if I try to use Jane’s key and certificate to login as, say, ansible on the target node, that fails because Jane’s user key certificate contains a principal “jane” only. (Add more comma-separated principal names when signing the user key in the -n option)
$ssh -i jane -l ansible alice.example.com
sshd-session[3648]: error: Certificate invalid: name is not a listed principal
a user’s certificate can be forced to invoke a particular utility on the server instead of what the user wanted. Here we add a forced date, and in spite of the user wanting to invoke uname, date is executed on the remote node.
$ssh-keygen -s CA/ssh-ca -I"Jane Jolie"-n jane -z 2 -V +1w -O force-command=/usr/bin/date jane.pub
$ssh -i jane -l jane alice.example.com uname Fri Mar 27 13:43:58 UTC 2026
permissible source CIDR masks can be embedded into a user certificate, and the server logs violations
$ssh-keygen -s CA/ssh-ca -I"Jane Jolie"-n jane -z 3 -V +1w -O force-command=/usr/bin/date -O source-address=192.0.2.0/24 jane.pub
sshd-session[3854]: cert: Authentication tried for jane with valid certificate but not from a permitted source address (192.168.1.100).
it’s worth looking at the description of AuthorizedPrincipals in which I can specify a file that lists principal names accepted for certificate authentication; there is also AuthorizedPrincipalsCommand with which I can specify a command which retrieves the list from an arbitrary source, similar to what we did with AuthorizedKeysCommand.
Revocation
Certificates can expire as we saw above, but sometimes we want to actually revoke a key, which we can do with an revocation file which contains any number of revoked keys.
I create this file on my CA machine:
$ssh-keygen -k-f revoked jane.pub
Revoking from jane.pub
$file revoked
revoked: OpenSSH key/certificate revocation list, format 1, version 0, generated Tue Apr 7 14:24:33 2026
I install the revoked file on the target node and configure it in sshd_config:
Key revocation lists can contain a complex set of rules specifying which keys are revoked. See the KEY REVOCATION LISTS section in [ssh-keygen(1)20.
Checklist
I’ve tried to assemble a bit of a checklist of things we verify particularly if something doesn’t work as expected.
on the node:
CA’s public key must be on server, readable by sshd, and configured in TrustedUserCAKeys
server’s host key needs signing and must be placed alongside host key file(s)
sshd_config requires HostCertificate pointing to the certificate of the host key
server needs restarting
if the node’s certificate changes, the service needs to be restarted
on the client:
user’s SSH key needs to be signed by CA and placed adjacent to the key file
user’s SSH key may be in SSH agent; if it isn’t and the key is encrypted, SSH will as usual ask for its passphrase if there is one
known_hosts needs correct @cert-authority; pay attention to principal names (test with * to permit any hostname)
the node name or address used when connecting to server must match one of the principal names in host key certificate on server
If I do get prompted to confirm the host fingerprint, it is possible the certificate has expired, and I am told so.
$ssh -oStrictHostKeyChecking=ask -oIdentitiesOnly=true-i jane -l root 192.0.2.65
Certificate invalid: expired
The authenticity of host '192.0.2.65 (192.0.2.65)' can't be established.
ED25519 key fingerprint is SHA256:3zm2UIyqO0xsxejndKdg9+Rpm4JeGuxamlsZwNOGX2Y.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? ^C
on the other hand, if I’m prompted for a password, it is quite likely that a constraint in the certificate cannot be validated. Here I’m tyring to login as a user for which the certificate has no principal configure. The server log shows it.
$ssh -i jane -l ansible 192.0.2.65
ansible@192.0.2.65's password:
% journalctl -f
sshd-session[6622]: error: Certificate invalid: name is not a listed principal
if the server had been configured to permit only public key authentication (no passwords), Jane would have seen the following diagnostic and not had the possibility to enter a password.
$ssh -i jane -l ansible 192.0.2.65
ansible@192.0.2.65: Permission denied (publickey).
We accomplish this by configuring /etc/ssh/sshd_config as follows:
PubkeyAuthentication yes
PasswordAuthentication no
ChallengeResponseAuthentication no
adding a user’s key to an SSH agent shows that both it and the cert have been added (both need to be removed with -d if desired)
$ssh-add jane
Enter passphrase for jane:
Identity added: jane (Jane's key)
Certificate added: jane-cert.pub (Jane Jolie)
$ssh-add -l 256 SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE Jane's key (ECDSA)
256 SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE Jane's key (ECDSA-CERT)
during signing of user certificates features can be enabled or disabled. Here we clear all options, then add permission for agent forwarding and port forwarding. Note that pseudo-tty (PTY) allocation is therefore explicitly disabled. We can login to the target node, but do not get a PTY allocated, i.e. we have no console
$ssh-keygen -U\-s CA/ssh-ca \-I"Jane Jolie"\-n jane,root \-z 4 \-V +1w \-O clear \-O extension:permit-agent-forwarding \-O extension:permit-port-forwarding \
jane.pub
Signed user key jane-cert.pub: id "Jane Jolie" serial 4 for jane,root valid from 2026-03-28T15:01:00 to 2026-04-04T16:02:02
$ssh-keygen -L-f jane-cert.pub
jane-cert.pub:
Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
Public key: ECDSA-CERT SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE
Signing CA: ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM (using ecdsa-sha2-nistp256)
Key ID: "Jane Jolie"
Serial: 4
Valid: from 2026-03-28T15:01:00 to 2026-04-04T16:02:02
Principals:
jane
root
Critical Options: (none)
Extensions:
permit-agent-forwarding
permit-port-forwarding
$ssh -l jane o78
PTY allocation request failed on channel 0
$ssh -l jane o78 uname OpenBSD
And the “magical” part of all this? The signing user’s SSH certificate configures these capabilities: neither the key, nor an entry in a user’s authorized_keys file on the target system have been added.
Automate host key certificate distribution?
Being able to easily sign host keys got me thinking that it would likely be easy’ish to do this automatically. I quickly shredded my idea of an CGI (Common Gateway Interface) shell script which invoked ssh-keygen when I came accross sshkey-tools, a well-documented Python module with which I can programmatically issue and sign SSH certificates.
An initial sign.py was quickly created, and I then wrapped that into a BottlePy HTTP server, as proof of concept:
on the machine containing the CA, I launch my small host key bot (hkbot.py) for which I don’t yet have a logo but patent is pending. :-)
$ ./hkbot.py
Bottle v0.13.4 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8870/
Hit Ctrl-C to quit.
on a client node onto which I want to have a host key signed, I upload one of host’s public keys to the bot which will determine the key type (the Python module does that automatically), and sign a certificate which it returns together with very basic shell commands to install the CA’s public key, the newly issued certificate, and with which to patch /etc/ssh/sshd_config with the necessary statements.
# curl -sSf -F hostkey=@/etc/ssh/ssh_host_ed25519_key.pub http://192.0.2.140:8870 | sh
Extracting CA public key to /etc/ssh/ssh-ca.pub
Extracting certificate to /etc/ssh/ssh_host_ed25519_key-cert.pub
Patching /etc/ssh/sshd_config
Validating /etc/ssh/sshd_config
after verifying sshd_config I restart the service
# systemctl restart sshd
I should now be able to seamlessly login to the machine. Using -v to show what the server sends, and explicitly setting ask to make sure I’m informed should a host key not be verifiable.
$ ssh -v -o IdentitiesOnly=true -i jane -l root 192.0.2.65
...
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:4WTRnq2OR1m03TpnHCfkFdlh1gN/PBXE4vDi0WnjFEc, serial 27 ID "deadbeef01" CA ecdsa-sha2-nistp256 SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM valid from 2026-04-02T10:19:46 to 2026-04-02T11:19:46
...
root@node:~#
I keep an eye on the auth log (syslog) or the journal (systemd) for incoming connections on the machine
Apr 02 10:22:07 d13 sshd-session[1058]: Accepted publickey for root from 192.0.2.140 port 54872 ssh2: ECDSA-CERT SHA256:2WH263LauVfk5XvxHvrdvNt0y9OgaHBLSvcQ+R6u/KE ID Jane Jolie (serial 4) CA ECDSA SHA256:A5ZBb5b/GbAv03EAb8fmDzv4p+q0g8Ulxrt8QZpbamM
It’s a bit primitive, and there are lots of bits missing for real-life, but I was able to scratch an itch. I later scratched a further itch when deploying SSH host keys and certificates with Ansible, generating keys on the node and signing certificates on the controller.
The title of this post is “SSH certificates: the better SSH experience”, and I really believe SSH certificates provide the better and more seamless operation for environments in which I have control over the nodes:
No more need for TOFU (Trust On First Use), as clients and servers implicitly trust each other.
SSH certificates allow me to issue short-lived keys to users (you may login in the next 20 minutes, after that not again). (If the user remains logged in, there’s no automation to kick them out.)
The use of SSH certificates means I don’t have to clean up and remove public user keys from servers (i.e. remove entries from authorized_keys files); once the validity of the certificate expires, the user is locked out.
Certificates can contain forced-commands, we can permit or deny creation of PTY on the remote, and the target users (principals) and permissible source addresses can be specified.
I demonstrated above that some of this can be automated, and there is a very interesting project called Smallstep SSH with tools to enable all that and much more.
This is neat for hosts that you own, e.g. where you have control of the CA and access to its private key which one needs for signing hosts/user keys. The “traditional” known_hosts/authorized keys was made for multi-user systems where you want secure connections between systems where you do not have root privileges. But nowadays, most usage falls into the first category, I assume. And a CA makes that easier.
I learn from Job Snijders about an I-D for the SSH Certificate Format, in draft-miller-ssh-cert-06.
When I first wrote about discovering Netbox DNS I mentioned that the combination of Netbox and the DNS plugin don’t actually create a DNS server and zone transfers directly from Netbox are thus impossible. I also went on to describe how there exist programs which can export the data into, say, zone master files for further provisioning DNS servers, but the times, they are changing.
Sven Luethi has created a bridge between Peter Eckel’s netbox-plugin-dns, the NetBox plugin for managing DNS data, and the DNS. It is called netbox-plugin-dns-bridge (previously netbox-plugin-bind-provisioner) and implements a small DNS server within Netbox which provides for zone transfers (AXFR) in order for compliant clients to transfer whole zones. (Naming things is hard in IT, as we sometimes jokingly say, and I’d like to specifically point out that this is not just for a BIND name server – any compliant name server can transfer zones from netbox-plugin-dns-bridge.)
Setting this up is quite straightforward, and is well described in the README. I generate a TSIG key which I configure into the plugin.
What this does is to configure the provisioner to provide transfers for zones in the "external" view when a transfer is requested with the specified TSIG key. I can configure any number of views as long as I use distinct TSIG keys for each; this is the only way the mini DNS server can determine from which view it should serve a zone (or zones).
After restarting Netbox I test a zone transfer using dig(1). The TCP port of the mini DNS server is configurable, but I’ve used the default here. (Note also how the port is specified in the example named.conf stanza below.) I specify the filename containing the TSIG key, the IP address of the mini DNS server and request an AXFR.
$dig -k tsig.pext -p 5354 @127.0.0.1 example.net AXFR +noall +answer +onesoa +multi
example.net. 3608 IN SOA nsa.example. noc.example.org. (
1768034701 ;serial
7200 ;refresh (2 hours) 1800 ;retry (30 minutes) 4838400 ;expire (8 weeks) 900 ;minimum (15 minutes) )
alice.example.net. 3603 IN A 192.0.2.101
example.net. 3603 IN NS usb.example.
example.net. 3603 IN NS nsa.example.
goofy.example.net. 3603 IN A 192.0.2.102
minnie.example.net. 3603 IN A 192.0.2.103
The transfer works, so I can configure my DNS name server, here BIND, accordingly:
include "_demo/tsig.pext";
options {
...
masterfile-format text;
recursion no;
};
zone "example.net" in {
type secondary;
file "example.net";
primaries {
192.0.2.4 port 5354 key "pext";
};
};
Such a setup would suffice if I have just a small number of zones I want to transfer from Netbox DNS to my (primary) DNS server.
But what if we have many zones to transfer? Enter Catalog Zones.
Catalog Zones
In addition to providing AXFR for zones in particular views, netbox-plugin-dns-bridge provides support for Catalog Zones (RFC 9432) which are a method for provisioning DNS servers; Catalog Zone support exists in BIND, in NSD, in Knot-DNS, and in PowerDNS. The catalog is generated on-the-fly with member zones obtained from the view associated with the TSIG key used in the transfer. Member zone names are constructed from the base32-encoded bytes of a UUID4 stored with the django zone object, truncated to 26 characters in order to remove padding.
$dig -k tsig.pext -p 5354 @127.0.0.1 external.catz AXFR +noall +answer +onesoa +multi
external.catz. 0 IN SOA invalid. invalid. (
8 ;serial
60 ;refresh (1 minute) 10 ;retry (10 seconds) 1209600 ;expire (2 weeks) 0 ;minimum (0 seconds) )
2ytykijfebgjdjf4cz7xrmftta.zones.external.catz. 0 IN PTR 2.0.192.in-addr.arpa.
54aed5irbbabfhhmv2nrotj5mu.zones.external.catz. 0 IN PTR aaa.aa.
e5alhvgaq5bs7bvlq2fn7gq3fq.zones.external.catz. 0 IN PTR bbb.aa.
6ruw3o4merc6bjhsz6mv53wiq4.zones.external.catz. 0 IN PTR example.net.
external.catz. 0 IN NS invalid.
version.external.catz. 0 IN TXT "2"
Support for Catalog Zones might well be coming natively to netbox-plugin-dns, and Sven has already said:
I will coordinate with him [Peter Eckel] to ensure compatibility is not affected. Likely scenario will be a timely coordinated switch so that both plugins can be updated together and the DNS plugin takes over while the bind provisioner removes its catalog zone handling.
Configuring this, again using BIND as an example, is straightforward: I configure catalog-zones support in which I specify the name of the catalog zone (either catz or <viewname>.catz) and configure default primaries for the catalog members, followed by the zone stanza for the catalog zone proper:
options {
...
allow-new-zones yes;
catalog-zones {
zone "external.catz"
zone-directory "netbox-zones"
in-memory no
default-primaries { 192.0.2.4 port 5354 key "pext"; };
};
};
zone "external.catz" in {
type secondary;
file "external.catz";
primaries {
192.0.2.4 port 5354 key "pext";
};
};
One rndc reconfig later, I have my member zones and BIND begins serving these.
$ls-l netbox-zones/
-rw-r--r-- 1 jpm staff 457 Jan 23 15:31 __catz___default_external.catz_2.0.192.in-addr.arpa.db
-rw-r--r-- 1 jpm staff 593 Jan 23 15:31 __catz___default_external.catz_aaa.aa.db
-rw-r--r-- 1 jpm staff 408 Jan 23 15:31 __catz___default_external.catz_bbb.aa.db
-rw-r--r-- 1 jpm staff 419 Jan 23 15:31 __catz___default_external.catz_example.net.db
This is a really useful addition to Netbox DNS, with one thing missing: NOTIFY. There’s no provision for performing DNS NOTIFY from Netbox to the first nameserver, but a Webhook or Netbox script which invokes something like rndc retransfer in BIND (or equivalent for distinct name servers) would likely do the trick. Alternatively, setting the SOA REFRESH value to a small number will also work. On the other hand, Sven indicates he’s thinking about it, so we might be in for an implementation.
It is the year 2021, and I use an Albert Heijn shopping bag, one of the old ones obtained years earlier, when the ultimate nightmare occurs: upon leaving the supermarket I hear a ripping sound and the side of the bag tears open.
I post about the disaster and my distress on social media, but I am actually amused: I’d normally have no qualms in just ditching the bag and using something else. Somehow I’m attached to this awfully ugly blue bag.
My friends Ton from the Netherlands and Nicolai from Berlin promise to come to my aid and both send me a few Albert Heijn bags. I have never had the heart to confess to them that I didn’t actually use the bags; they had to pay for them and the postage to me. I still have them in the cellar, unused.
Meanwhile, you see, ah had redesigned their shopping bags. It becomes taller and narrower, and it looks as though it won’t take the loads I’m used to transporting: groceries for a week, possibly with tins and bottles. Certainly many kilograms of weight per load.
I see two possible solutions: I starve from no longer being able to shop for groceries or patch the bag, and thanks to the availability of a strip from a roll of duct tape I decide on the latter. It’s the strip along the height of the bag seen here on the bottom of the photo.
That patch held for almost five years!
I went grocery shopping today again, and added lots of heavy items to the bag: a cauliflower and other vegetables, a kg of meat, three 1.5L bottles of cleaning material, four tins of beer, a bottle of olive oil, several jars of capers, two tins of tomato polpo and three of tuna fish, etc. etc. and I suddenly feel more than hear a tear in the handles.
I make it to the self-checkout, and carefully repack the Albert Heijn bag. I gingerly carry it to the car. Upon arriving home I apply several strips of duct tape along the edges of the carrying handles, hoping to give the bag another breath of life.