SSH with Kubernetes: Making Permissions Follow Dynamic Workloads.
May 8, 2017 by Ev Kontsevoy
In the previous article of this series about managing Kubernetes, we explained how SSH clients can be authenticated using SSH certificates as opposed to simple public keys or passwords.
In this post, we’ll take the next step and use SSH certificates to implement a simple and effective role-based access control, where SSH users are granted access NOT to specific servers, but to specific data or workloads. This style of access control is more compatible with the dynamic nature of datacenter-level work scheduling.
This blog post will teach you:
- How to configure Kubernetes users to SSH into servers using Kubernetes labels instead of hostnames or IP addresses.
- How to restrict SSH permissions based on Kubernetes labels and user identities.
ssh [email protected]=db-master
db-master is a Kubernetes label dynamically applied to whatever machine is running
the master PosgtreSQL instance.
Why SSH into Clusters?
If you believe that you don’t need to ever “SSH into a box”, perhaps this article isn’t for you. The promise of simply throwing (scheduling) your code onto an intelligent “mainframe” and have it run by itself is appealing, but the need to SSH into a machine is still a common requirement, even if just for automation tooling.
Sample Use Case
Suppose you are in charge of a small ops team which is tasked with the job of managing a Kubernetes application called “Cookbook”. This application consists of two types of Kubernetes pods:
Worker: these handle HTTP requests from the clients and provide CRUD access to the database of recipes. They are stateless, numerous, cheap and can be easily moved around.
Database: these pods are responsible for actually storing the recipes, manage backups, restore, etc. Let’s say PostgreSQL is used for it.
Your ops team is composed of only two people: Elliot and Darlene. Both are capable of administering the workers but only Darlene knows how to manage the PostgreSQL database.
Both Darlene and Elliot can assume the role of
only Darlene can assume the role of
dba. Given his lack of knowledge, perhaps
you don’t even want to let Elliot access servers where “Database” pods are running.
Lets see how can this be done.
Access Follows Data
First, we need to make sure that Darlene or Elliot lose access to everything if they quit the organization. We covered how to do this in the previous article: by using auto-expiring certificates.
Wouldn’t it be convenient and more secure, if Darlene and Elliot would gain or lose SSH access to a
server based on where Kubernetes schedules the database to run? This way if a
database pod is
scheduled to run on server “A”, Darlene gains access to it (but not Elliot) and
both of them would be able to SSH into any server running a worker
The access should be dynamically granted/revoked to servers based on the workloads they are running.
For this to work we need the following to happen:
- SSH Certificate Authority (CA) needs to know that Darlene is
dbarole needs to be included in Darlene’s certificate.
- The servers with “database” pods must only trust users with certificates
Lets see how this can be done.
Labels = Access
Teleport supports the ability to apply labels to any server it’s running on, regardless of either a node is managed by Kubernetes or not. Now we need to expose Kubernetes labels to become recognizable as SSH “hostnames”.
For this we’ll use a Teleport label command. A “label command” is a CLI command matched to a label. A Teleport node will execute at a given frequency and the standard output of that command will become a value for the label, for example:
# snippet from /etc/teleport.yaml ssh_service: commands: # teleport daemon will execute /usr/bin/readrole command every minute and # the output of it will be assigned to role - name: role command: ["/usr/bin/readrole", "--label=role"] period: 1m0s
You will have to implement
readrole command using a programming language of your choice,
in this example
readrole command takes a
--label argument, queries Kubernetes API for the
role label for the node it’s running on and prints the output into stdout, like this:
# executed on DB master machine: $ /usr/bin/readrole --label=role db-master # executed on a regular non-DB machine: $ /usr/bin/readrole --label=role worker
readrole present and configured to be a Teleport command label, let’s check if the
nodes are correctly reporting their labels into a Teleport cluster:
$ tsh ls Node Name Address Labels --------- ------- ------ worker-1.comp 10.100.2.241:3022 role=worker instance-type=c3.xlarge public-ipv4=184.108.40.206 database-2.comp 10.100.2.245:3022 role=db-master instance-type=c3.4xlarge
So now you can SSH into a node which has
$ tsh ssh [email protected]=db-master
Enforcing SSH Restrictions
Now, with Kubernetes labels working as SSH hostnames, how do we make sure that only the
approved group of users can SSH into the server with
We need to create a new SSH Role (lets call it “dba”) in Teleport. Let’s start by creating a Teleport Role description and saving it in a YAML file:
# Save it as dba.yaml kind: role version: v1 metadata: name: dba spec: # Admins can SSH as root into machines logins: [root] # Only admins can SSH into any instances labeled with 'db-maser' role: node_labels: 'role': 'db-master'
Add this role to Teleport Enterprise:
$ tctl upsert -f dba.yaml
And now only SSH users with
dba role will be able to SSH into servers with
exactly what we wanted to accomplish.
In this post you have learned how to collect Kubernetes labels for your nodes and report them into a Teleport Cluster, allowing you to SSH into the nodes using their labels instead of host names.
Additionally, you leanred how to restrict SSH access to your servers to specific users based on its nodes’ Kubernetes labels.
The end setup looks a bit clunky due to the need to implement an external label-reporting command but that’s because Teleport is implemented without hard dependencies on Kubernetes and it’s designed to work with any external infrastructure labeling scheme (like AWS).
For a more seamless integration, we have a CNCF certified Kubernetes distribution called Gravity which tightly integrates Teleport SSH underneath Kubernetes, among other things, which you may want to check out.