How to SSH Properly
There’s no denying that SSH is the de facto tool for *nix server administration. It’s far from perfect, but it was designed with security in mind and there’s been a huge amount of tooling written over the years to make it easier to use. In addition, many popular products and just about every server deployment system integrates with SSH somehow. It is universally supported across pretty much all architectures and distributions, from Raspberry Pis all the way up to massive supercomputer clusters.
SSH is a powerful tool which often grants a lot of access to anyone using it to log into a server. In this post, I’m going to talk about a few different ways that you can easily improve the security of your SSH model without needing to deploy a new application or make any huge changes to user experience.
Most people can agree that using public key authentication for SSH is generally better than using passwords. Nobody ever types in a private key, so it can’t be keylogged or observed over your shoulder. SSH keys have their own issues, however, some of which we’ve covered in a previous post about SSH key management.
The next level up from SSH keys is SSH certificates. OpenSSH has supported the use of certificates since OpenSSH 5.4 which was released back in 2010.
With SSH certificates, you generate a certificate authority (CA) and then use this to issue and cryptographically sign certificates which can authenticate users to hosts, or hosts to users. You can generate a keypair using the
ssh-keygen command, like this:
$ ssh-keygen -t rsa -b 4096 -f host_ca -C host_ca Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in host_ca. Your public key has been saved in host_ca.pub. The key fingerprint is: SHA256:tltbnMalWg+skhm+VlGLd2xHiVPozyuOPl34WypdEO0 host_ca The key's randomart image is: +---[RSA 4096]----+ | +o.| | .+..o| | o.o.+ | | o o.= E| | S o o=o | | ....+ = +.| | ..=. %.o.o| | *o Oo=.+.| | .oo=ooo+..| +----[SHA256]-----+ $ ls -l total 8 -rw-------. 1 gus gus 3381 Mar 19 14:30 host_ca -rw-r--r--. 1 gus gus 737 Mar 19 14:30 host_ca.pub
host_ca file is the host CA’s private key and should be protected. Don’t give it out to anyone, don’t copy it anywhere, and make sure that as few people have access to it as possible. Ideally, it should live on a machine which doesn’t allow direct access and all certificates should be issued by an automated process.
In addition, it’s best practice to generate and use two separate CAs - one for signing host certificates, one for signing user certificates. This is because you don’t want the same processes that add hosts to your fleet to also be able to add users (and vice versa). Using separate CAs also means that in the event of a private key being compromised, you only need to reissue the certificates for either your hosts or your users, not both at once.
As such, we’ll also generate a
user_ca with this command:
$ ssh-keygen -t rsa -b 4096 -f user_ca -C user_ca
user_ca file is the user CA’s private key and should also be protected in the same way as the host CA’s private key.
Issuing host certificates (to authenticate hosts to users)
In this example, we’ll generate a new host key (with no passphrase), then sign it with our CA. You can also sign the existing SSH host public key if you’d prefer not to regenerate a new key by copying the file (
/etc/ssh/ssh_host_rsa_key.pub) from the server, signing it on your CA machine, and then copying it back.
$ ssh-keygen -f ssh_host_rsa_key -N '' -b 4096 -t rsa $ ls -l -rw------- 1 ec2-user ec2-user 3247 Mar 17 14:49 ssh_host_rsa_key -rw-r--r-- 1 ec2-user ec2-user 764 Mar 17 14:49 ssh_host_rsa_key.pub $ ssh-keygen -s host_ca -I host.example.com -h -n host.example.com -V +52w ssh_host_rsa_key.pub Enter passphrase: # the passphrase used for the host CA Signed host key ssh_host_rsa_key-cert.pub: id "host.example.com" serial 0 for host.example.com valid from 2020-03-16T15:00:00 to 2021-03-15T15:01:37 $ ls -l -rw------- 1 ec2-user ec2-user 3247 Mar 17 14:49 ssh_host_rsa_key -rw-r--r-- 1 ec2-user ec2-user 2369 Mar 17 14:50 ssh_host_rsa_key-cert.pub -rw-r--r-- 1 ec2-user ec2-user 764 Mar 17 14:49 ssh_host_rsa_key.pub
ssh_host_rsa_key-cert.pub contains the signed host certificate.
Here’s an explanation of the flags used:
-s host_ca: specifies the filename of the CA private key that should be used for signing.
-I host.example.com: the certificate’s identity - an alphanumeric string that will identify the server. I recommend using the server’s hostname. This value can also be used to revoke a certificate in future if needed.
-h: specifies that this certificate will be a host certificate rather than a user certificate.
-n host.example.com: specifies a comma-separated list of principals that the certificate will be valid for authenticating - for host certificates, this is the hostname used to connect to the server. If you have DNS set up, you should use the server’s FQDN (for example
host.example.com) here. If not, use the hostname that you will be using in an
~/.ssh/configfile to connect to the server.
-V +52w: specifies the validity period of the certificate, in this case 52 weeks (one year). Certificates are valid forever by default - expiry periods for host certificates are highly recommended to encourage the adoption of a process for rotating and replacing certificates when needed.
Configuring SSH to use host certificates
You also need to tell the server to use this new host certificate. Copy the three files you just generated to the server, store them under the
/etc/ssh directory, set the permissions to match the other files there, then add this line to your
Once this is done, restart
systemctl restart sshd.
Your server is now configured to present a certificate to anyone who connects. For your local
ssh client to make use of this (and automatically trust the host based on the certificate’s identity), you will also need to add the CA’s public key to your
You can do this by taking the contents of the
host_ca.pub file, adding
@cert-authority *.example.com to the beginning, then appending the contents to
@cert-authority *.example.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDwiOso0Q4W+KKQ4OrZZ1o1X7g3yWcmAJtySILZSwo1GXBKgurV4jmmBN5RsHetl98QiJq64e8oKX1vGR251afalWu0w/iW9jL0isZrPrmDg/p6Cb6yKnreFEaDFocDhoiIcbUiImIWcp9PJXFOK1Lu8afdeKWJA2f6cC4lnAEq4sA/Phg4xfKMQZUFG5sQ/Gj1StjIXi2RYCQBHFDzzNm0Q5uB4hUsAYNqbnaiTI/pRtuknsgl97xK9P+rQiNfBfPQhsGeyJzT6Tup/KKlxarjkMOlFX2MUMaAj/cDrBSzvSrfOwzkqyzYGHzQhST/lWQZr4OddRszGPO4W5bRQzddUG8iC7M6U4llUxrb/H5QOkVyvnx4Dw76MA97tiZItSGzRPblU4S6HMmCVpZTwva4LLmMEEIk1lW5HcbB6AWAc0dFE0KBuusgJp9MlFkt7mZkSqnim8wdQApal+E3p13d0QZSH3b6eB3cbBcbpNmYqnmBFrNSKkEpQ8OwBnFvjjdYB7AXqQqrcqHUqfwkX8B27chDn2dwyWb3AdPMg1+j3wtVrwVqO9caeeQ1310CNHIFhIRTqnp2ECFGCCy+EDSFNZM+JStQoNO5rMOvZmecbp35XH/UJ5IHOkh9wE5TBYIeFRUYoc2jHNAuP2FM4LbEagGtP8L5gSCTXNRM1EX2gQ== host_ca
*.example.com is a pattern match, indicating that this certificate should be trusted for identifying any host which you connect to that has a domain of
*.example.com - such as
host.example.com above. This is a comma-separated list of applicable hostnames for the certificate, so if you’re using IP addresses or SSH config entries here, you can change this to something like
22.214.171.124,126.96.36.199 as appropriate.
Once this is configured, remove any old host key entries for
host.example.com in your
~/.ssh/known_hosts file, and start an
ssh connection. You should be connected straight to the host without needing to trust the host key. You can check that the certificate is being presented correctly with a command like this:
$ ssh -vv host.example.com 2>&1 | grep "Server host certificate" debug1: Server host certificate: [email protected] SHA256:dWi6L8k3Jvf7NAtyzd9LmFuEkygWR69tZC1NaZJ3iF4, serial 0 ID "host.example.com" CA ssh-rsa SHA256:8gVhYAAW9r2BWBwh7uXsx2yHSCjY5OPo/X3erqQi6jg valid from 2020-03-17T11:49:00 to 2021-03-16T11:50:21 debug2: Server host certificate hostname: host.example.com
At this point, you could continue by issuing host certificates for all hosts in your estate using your host CA. The benefit of doing this is twofold: you no longer need to rely on the insecure trust on first use (TOFU) model for new hosts, and if you ever redeploy a server and therefore change the host key for a certain hostname, your new host could automatically present a signed host certificate and avoid the dreaded
WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! message.
Issuing user certificates (to authenticate users to hosts)
In this example, we’ll generate a new user key and sign it with our user CA. It’s up to you whether you use a passphrase or not.
$ ssh-keygen -f user-key -b 4096 -t rsa $ ls -l -rw-r--r--. 1 gus gus 737 Mar 19 16:33 user-key.pub -rw-------. 1 gus gus 3369 Mar 19 16:33 user-key $ ssh-keygen -s user_ca -I [email protected] -n ec2-user,gus -V +1d user-key.pub Enter passphrase: # the passphrase used for the user CA Signed user key user-key-cert.pub: id "[email protected]" serial 0 for ec2-user,gus valid from 2020-03-19T16:33:00 to 2020-03-20T16:34:54 $ ls -l -rw-------. 1 gus gus 3369 Mar 19 16:33 user-key -rw-r--r--. 1 gus gus 2534 Mar 19 16:34 user-key-cert.pub -rw-r--r--. 1 gus gus 737 Mar 19 16:33 user-key.pub
user-key-cert.pub contains the signed user certificate. You’ll need both this and the private key (
user-key) for logging in.
Here’s an explanation of the flags used:
-s user_ca: specifies the CA private key that should be used for signing
-I [email protected]: the certificate’s identity, an alphanumeric string that will be visible in SSH logs when the user certificate is presented. I recommend using the email address or internal username of the user that the certificate is for - something which will allow you to uniquely identify a user. This value can also be used to revoke a certificate in future if needed.
-n ec2-user,gus: specifies a comma-separated list of principals that the certificate will be valid for authenticating, i.e. the *nix users which this certificate should be allowed to log in as. In our example, we’re giving this certificate access to both
-V +1d: specifies the validity period of the certificate, in this case
+1dmeans 1 day. Certificates are valid forever by default, so using an expiry period is a good way to limit access appropriately and ensure that certificates can’t be used for access perpetually.
If you need to see the options that a given certificate was signed with, you can use
$ ssh-keygen -L -f user-key-cert.pub user-key-cert.pub: Type: [email protected] user certificate Public key: RSA-CERT SHA256:egWNu5cUZaqwm76zoyTtktac2jxKktj30Oi/ydrOqZ8 Signing CA: RSA SHA256:tltbnMalWg+skhm+VlGLd2xHiVPozyuOPl34WypdEO0 (using ssh-rsa) Key ID: "[email protected]" Serial: 0 Valid: from 2020-03-19T16:33:00 to 2020-03-20T16:34:54 Principals: ec2-user gus Critical Options: (none) Extensions: permit-X11-forwarding permit-agent-forwarding permit-port-forwarding permit-pty permit-user-rc
Configuring SSH for user certificate authentication
Once you’ve signed a certificate, you also need to tell the server that it should trust certificates signed by the user CA. To do this, copy the
user_ca.pub file to the server and store it under
/etc/ssh, fix the permissions to match the other public key files in the directory, then add this line to
Once this is done, restart
systemctl restart sshd.
Your server is now configured to trust anyone who presents a certificate issued by your user CA when they connect. If you have a certificate in the same directory as your private key (specified with the
-i flag, for example
ssh -i /home/gus/user-key [email protected]), it will automatically be used when connecting to servers.
If you look in your server’s
sshd log (for example, by running
journalctl -u sshd), you will see the name of the certificate being used for authentication, along with the fingerprint of the signing CA:
sshd: Accepted publickey for ec2-user from 188.8.131.52 port 53734 ssh2: RSA-CERT ID [email protected] (serial 0) CA RSA SHA256:tltbnMalWg+skhm+VlGLd2xHiVPozyuOPl34WypdEO0
If the user tries to log in as a principal (user) which they do not have permission to use (for example, their certificate grants
ec2-user but they try to use
root), you’ll see this error in the logs:
sshd: error: key_cert_check_authority: invalid certificate sshd: error: Certificate invalid: name is not a listed principal
If the certificate being presented has expired, you’ll see this error in the logs:
sshd: error: key_cert_check_authority: invalid certificate sshd: error: Certificate invalid: expired
One way that you could make further use of user certificates is to set up a script which will use your CA to issue a certificate to log into production servers, valid only for the next two hours. Every use of this script or process could add logs as to who requested a certificate and embed their email address into the certificate. After this is done, every time the user uses that certificate to access a server (regardless of which principal they log in as), their email address will be logged. This can provide a useful part of an audit trail.
There are many other useful things you can do with SSH certificates, such as forcing a specific command to be run when presenting a certain certificate, or denying the ability to forward ports, X11 traffic or SSH agents. Take a look at
man ssh-keygen for some ideas.
Enforce the use of a bastion host
Another way to improve your SSH security is to enforce the use of a bastion host - a server which is specifically designed to be the only gateway for access to your infrastructure. Lessening the size of any potential attack surface through the use of firewalls enables you to keep a better eye on who is accessing what.
Switching to the use of a bastion host doesn’t have to be an arduous task, especially if you’re using SSH certificates. By setting up your local
~/.ssh/config file, you can automatically configure all connections to hosts within a certain domain to go through the bastion.
Here’s a very quick example of how to force SSH access to any host in the example.com domain to be routed through a bastion host,
Host *.example.com ProxyJump bastion.example.com IdentityFile ~/user-key Host bastion.example.com ProxyJump none IdentityFile ~/user-key
To make this even simpler, if you add
user-key to your local
ssh-add user-key, you can also remove the
IdentityFile entries from the SSH config file.
Once you’re using the bastion host for your connections, you can use
iptables (or another *nix firewall configuration tool of your choosing) on servers behind the bastion to block all other incoming SSH connections. Here’s a rough example using
$ iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT $ iptables -A INPUT -p tcp --dport 22 -s <public IP of the bastion> -j ACCEPT $ iptables -A INPUT -p tcp --dport 22 -j DROP
It’s a good idea to leave a second SSH session connected to the bastion while running these commands so that if you inadvertently input the wrong IP address or command, you should still have working access to the bastion to fix it via the already-established connection.
Add 2-factor authentication to your SSH logins
2-factor authentication makes it more difficult for bad actors to log into your systems by enforcing the need for two different “factors” or methods to be able to successfully authenticate.
This usually comes down to needing both “something you know” (like a password, or SSH certificate in our example) and “something you have” (like a token from an app installed on your phone, or an SMS with a unique code). One other possibility is requiring the use of “something you are” - for example a fingerprint, or your voice.
In this example, we’ll install the
google-authenticator pluggable authentication module, which will require users to input a code from the Google Authenticator app on their phone in order to log in successfully. You can download the app for iOS here and Android here.
As a general note, it’s always important to consider the user experience when enforcing security measures. If your measures are too draconian then users may attempt to find ways to defeat and work around them, which will eventually reduce the overall security of your systems and lead to the creation of back doors. To give our users a reasonable experience in this example, we are only going to require 2-factor authentication to be able to log into our bastion host. Once authenticated there, users should be able to log into other hosts simply by using their valid SSH certificate. This combination should give an acceptable level of security without interfering too much with user workflows. With this in mind, however, it is always prudent and appropriate to enforce extra security measures in specific environments which contain critical production data or sensitive information.
On RHEL/CentOS based systems, you can install the
google-authenticator module from the EPEL repository:
$ sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm # for RHEL/CentOS 7, change for other versions $ sudo yum install google-authenticator
For Debian/Ubuntu-based systems, this is available as the
$ sudo apt-get install libpam-google-authenticator
google-authenticator module has many options you can set which are documented here. In the interest of saving time, we are going to use some sane defaults in this example: disallow reuse of the same token twice, issue time-based rather than counter-based codes, and limit the user to a maximum of three logins every 30 seconds. To set up Google 2-factor authentication with these settings, a user should run this command:
$ google-authenticator -d -f -t -r 3 -R 30 -W
You can also run
google-authenticator with no flags and answer some prompts to set up interactively if you prefer.
This will output a QR code that the user can scan with the app on their phone, plus some backup codes which they can use if they lose access to the app. These codes should be stored offline in a secure location.
Scan the generated QR code for your user now with the Google Authenticator app and make sure that you have a 6-digit code displayed. If you need to edit or change any settings in future, or remove the functionality completely, the configuration will be stored under
Configure PAM for 2-factor authentication
To make the system enforce the use of these OTP (one-time password) codes, we’ll first need to edit the PAM configuration for the
sshd service (
/etc/pam.d/sshd) and add this line to the end of the file:
auth required pam_google_authenticator.so nullok
nullok at the end of this line means that users who don’t have a second factor configured yet will still be allowed to log in so that they can set one up. Once you have 2-factor set up for all your users, you should remove
nullok from this line to properly enforce the use of a second factor.
We also need to change the default authentication methods so that SSH won’t prompt users for a password if they don’t present a 2-factor token. These changes are also made in the
- On RHEL/CentOS-based systems, comment out
auth substack password-authby adding a
#to the beginning of the line:
#auth substack password-auth
- On Debian/Ubuntu-based systems, comment out
@include common-authby adding a
#to the beginning of the line:
/etc/pam.d/sshd file once you’re done.
Configure SSH for 2-factor authentication
We also need to tell SSH to require the use of 2-factor authentication. To do this, we make a couple of changes to the
Firstly, we need to change
ChallengeResponseAuthentication no to
ChallengeResponseAuthentication yes to allow the use of PAM for credentials.
We also need to set the list of acceptable methods for authentication by adding this line to the end of the file (or editing the line if it already exists):
This tells SSH that it should require both a public key (which we are going to be satisfying using an SSH certificate) and a keyboard-interactive authentication (which will be provided and satisfied by the
sshd PAM stack). Save the
/etc/ssh/sshd_config file once you’re done.
At this point, you should restart
systemctl restart sshd. Make sure to leave an SSH connection open so that you can fix any errors if you need to. Restarting SSH will leave existing connections active, but new connections may not be allowed if there is a configuration problem.
Test it out
Connect to your bastion host directly and you should see a prompt asking you for your 2-factor code:
$ ssh bastion.example.com Verification code:
Type the code presented by your Google Authenticator app and your login should proceed normally.
If you check the
sshd log with
journalctl -u sshd, you should see a line indicating that your login succeeded:
Mar 23 16:51:13 ip-172-31-33-142.ec2.internal sshd: Accepted keyboard-interactive/pam for gus from 184.108.40.206 port 42622 ssh2
The methods above give practical examples of several ways in which you can improve the security of your SSH infrastructure, all while giving users the flexibility to keep using the tools they’re familiar with.ssh 2-factor teleport bastion certficates
Want to stay informed?
Subscribe to our blog to get articles and product updates.