Teleport

Set up Automatic Agent Upgrades

Teleport supports automatic agent upgrades for systemd-based Linux distributions using apt, yum, and zypper package managers, as well as Kubernetes clusters.

Teleport agents run an upgrader that queries a version server to determine whether they are out of date. This guide describes how to set up your infrastructure to support automatic updates. If you are a Teleport Cloud user or run a version server already, return to the Upgrading menu for the appropriate next steps to upgrade Teleport.

The Automatic Update Architecture guide explains how automatic agent upgrades work in more detail.

Warning

Systemd agents enrolled into automatic updates can only install versions present in their package repositories. As Teleport 14 won't be published to stable/v13, those agents will require manual intervention to be upgraded to the next major version (adding a new APT/YUM/zypper repo for stable/v14). Alternatively, you can use the stable/rolling channel, which contains Teleport v13.3.2 forward, including future major releases.

Prerequisites

Familiarity with the Upgrading Compatibility Overview guide, which describes the sequence in which to upgrade components of your cluster.
Self-hosted Teleport cluster running at least v14.3.7 or v15.1.3.
The tctl and tsh client tools version >= 15.2.4. Read Installation for how to install these.
To check that you can connect to your Teleport cluster, sign in with tsh login, then verify that you can run tctl commands using your current credentials. tctl is supported on macOS and Linux machines. For example:
```
tsh login --proxy=teleport.example.com --user=[email protected]
tctl status
Cluster  teleport.example.com
Version  15.2.4
CA pin   sha256:abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678abdc1245efgh5678
```
If you can connect to the cluster and run the tctl status command, you can use your current credentials to run subsequent tctl commands from your workstation. If you host your own Teleport cluster, you can also run tctl commands on the computer that hosts the Teleport Auth Service for full permissions.

Step 1/4. Check the version advertised by the Teleport Proxy Service

The Teleport Proxy service exposes by default a version channel called default that serves the same version the Proxy Service is running.

If your proxy public address is teleport.example.com:443, you can query the version server with the following command:

curl "https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/default/version"
15.2.4

By default, the version server has a single default channel, serving the version of the Teleport Proxy Service. If you want to override the default version or add other channels you can use the automatic_upgrades_channels field in the Proxy Service configuration file:

proxy_service:
  enabled: "yes"
  automatic_upgrades_channels:
    # Override the default version channel reachable at
    # https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/default/version
    default:
      static_version: v14.2.1
    # Define a new version channel with a static version reachable at
    # https://teleport.example.com:443/v1/webapi/automaticupgrades/channel/m-static-channel/version
    my-static-channel:
      static_version: v14.2.0
    # Define a new version channel forwarding requests to an upstream version server
    my-remote-channel:
      forward_url: https://updates.releases.teleport.dev/v1/stable/cloud

You must ensure all proxies share the same automatic_upgrades_channels configuration. If Proxy Service instances are configured differently, you will experience agents flickering between versions as the version served is not consistent across instances.

Step 2/4. Configure the maintenance schedule

At this point, the updaters can be configured to pull the version from the release channel and upgrade the agents. In this step you'll configure a maintenance schedule for the Teleport cluster that agents will use to determine when to check for upgrades.

Create a Teleport role

Create a Teleport role that can manage cluster maintenance configurations through the cluster_maintenance_config dynamic resource. No preset Teleport roles provide this ability, so you will need to create one.

Create a file called cmc-editor.yaml with the following content:

kind: role
version: v7
metadata:
  name: cmc-editor
spec:
  allow:
    rules:
    - resources: ['cluster_maintenance_config']
      verbs: ['create', 'read', 'update', 'delete']

Create the role resource:
```
tctl create cmd-editor.yaml
```
Add the role to your Teleport user:

Assign the myrole role to your Teleport user by running the appropriate commands for your authentication provider:

Retrieve your local user's roles as a comma-separated list:

ROLES=$(tsh status -f json | jq -r '.active.roles | join(",")')

Edit your local user to add the new role:

tctl users update $(tsh status -f json | jq -r '.active.username') \  --set-roles "${ROLES?},myrole"

Sign out of the Teleport cluster and sign in again to assume the new role.

Retrieve your github authentication connector:
```
tctl get github/github --with-secrets > github.yaml
```
Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the github.yaml file. Because this key contains a sensitive value, you should remove the github.yaml file immediately after updating the resource.
Edit github.yaml, adding myrole to the teams_to_roles section.

The team you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the team must include your user account and should be the smallest team possible within your organization.

Here is an example:
```
  teams_to_roles:
    - organization: octocats
      team: admins
      roles:
        - access
+       - myrole
```
Apply your changes:
```
tctl create -f github.yaml
```
Sign out of the Teleport cluster and sign in again to assume the new role.

Retrieve your saml configuration resource:
```
tctl get --with-secrets saml/mysaml > saml.yaml
```
Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the saml.yaml file. Because this key contains a sensitive value, you should remove the saml.yaml file immediately after updating the resource.
Edit saml.yaml, adding myrole to the attributes_to_roles section.

The attribute you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

Here is an example:
```
  attributes_to_roles:
    - name: "groups"
      value: "my-group"
      roles:
        - access
+       - myrole
```
Apply your changes:
```
tctl create -f saml.yaml
```
Sign out of the Teleport cluster and sign in again to assume the new role.

Retrieve your oidc configuration resource:
```
tctl get oidc/myoidc --with-secrets > oidc.yaml
```
Note that the --with-secrets flag adds the value of spec.signing_key_pair.private_key to the oidc.yaml file. Because this key contains a sensitive value, you should remove the oidc.yaml file immediately after updating the resource.
Edit oidc.yaml, adding myrole to the claims_to_roles section.

The claim you should map to this role depends on how you have designed your organization's role-based access controls (RBAC). However, the group must include your user account and should be the smallest group possible within your organization.

Here is an example:
```
  claims_to_roles:
    - name: "groups"
      value: "my-group"
      roles:
        - access
+       - myrole
```
Apply your changes:
```
tctl create -f oidc.yaml
```
Sign out of the Teleport cluster and sign in again to assume the new role.

Create a cluster maintenance configuration

Create the following cmc.yaml manifest allowing maintenance on Monday, Wednesday and Friday between 02:00 and 03:00 UTC.

kind: cluster_maintenance_config
spec:
  agent_upgrades:
    # Maintenance window start hour in UTC.
    # The maintenance window lasts 1 hour.
    utc_start_hour: 2
    # Week days when maintenance is allowed
    # Possible values are:
    # - Short names: Sun, Mon, Tue, Wed, Thu, Fri, Sat
    # - Long names: Sunday, Monday, Tuesday, Wednesday, Thursday, Friday, Saturday
    weekdays:
      - Mon
      - Wed
      - Fri

Finally, apply the manifest using tctl:

tctl create cmc.yaml
maintenance window has been updated

Step 3/4. Enroll Kubernetes agents in automatic updates

Now that you have deployed a version server, you can enroll agents in automatic upgrades. This guide begins with agents deployed on Kubernetes. If all of your agents run on Linux servers, you can skip to Step 4.

Install the agent upgrader Helm chart

This section assumes that the name of your teleport-kube-agent release is teleport-agent, and that you have installed it in the teleport namespace.

Add the following chart values to the values file for the teleport-kube-agent chart:

updater:
  enabled: true
  # No need to set the `versionServer` value anymore since v13.4.15, v14.3.1 and v15.
  releaseChannel: default

Update the Helm chart release with the new values:

helm -n teleport  upgrade teleport-agent teleport/teleport-kube-agent \--values=values.yaml \--version=15.2.4

Verify that the upgrader is working properly

You can validate the updater is running properly by checking if its pod is ready:

kubectl -n teleport-agent get pods
NAME                               READY   STATUS    RESTARTS   AGE<your-agent-release>-0                         1/1     Running   0          14m<your-agent-release>-1                         1/1     Running   0          14m<your-agent-release>-2                         1/1     Running   0          14m<your-agent-release>-updater-d9f97f5dd-v57g9   1/1     Running   0          16m

Check for any deployment issues by checking the updater logs:

kubectl -n teleport logs deployment/teleport-agent-updater
2023-04-28T13:13:30Z	INFO	StatefulSet is already up-to-date, not updating.	{"controller": "statefulset", "controllerGroup": "apps", "controllerKind": "StatefulSet", "StatefulSet": {"name":"my-agent","namespace":"agent"}, "namespace": "agent", "name": "my-agent", "reconcileID": "10419f20-a4c9-45d4-a16f-406866b7fc05", "namespacedname": "agent/my-agent", "kind": "StatefulSet", "err": "no new version (current: \"v12.2.3\", next: \"v12.2.3\")"}

Troubleshooting automatic agent upgrades on Kubernetes

The updater is a controller that periodically reconciles expected Kubernetes resources with those in the cluster. The updater executes a reconciliation loop every 30 minutes or in response to a Kubernetes event. If you don't want to wait until the next reconciliation, you can trigger an event.

Any deployment update will send an event, so you can trigger the upgrader by annotating the resource:
```
kubectl -n teleport annotate statefulset/teleport-agent 'debug.teleport.dev/trigger-event=1'
```
To suspend automatic updates for an agent, annotate the agent deployment with teleport.dev/skipreconcile: "true", either by setting the annotations.deployment value in Helm, or by patching the deployment directly with kubectl.

Step 4/4. Enroll Linux agents in automatic updates

This section shows you how to enroll Teleport agents running on Linux virtual or bare-metal machines into automatic updates.

Follow these instructions on each of your Teleport agents.

Install the agent upgrader

Ensure the Teleport repository is added and Teleport Enterprise is installed.

To verify if the Teleport repository was added to the system, check if either of the follow files exist:
```
ls /etc/apt/sources.list.d/teleport.list
or
ls /etc/yum.repos.d/teleport.repo
```
The upgrader checks the repository for available releases, so make sure that it is up to date.
If the repository was added, make sure the Teleport binary installed on the agent can run the automatic updater:
```
which teleport-upgrade || echo "Install the upgrader"
Install the upgrader
```

If the Teleport repository is not found, or the Teleport binary you installed does not include the upgrader, add the appropriate repository and reinstall Teleport.

Download Teleport's PGP public key
sudo curl https://apt.releases.teleport.dev/gpg \-o /usr/share/keyrings/teleport-archive-keyring.asc
Source variables about OS version
source /etc/os-release
Add the Teleport APT repository for v15. You'll need to update this
file for each major release of Teleport.
echo "deb [signed-by=/usr/share/keyrings/teleport-archive-keyring.asc] \https://apt.releases.teleport.dev/${ID?} ${VERSION_CODENAME?} stable/v15" \| sudo tee /etc/apt/sources.list.d/teleport.list > /dev/null

sudo apt-get update
sudo apt-get install teleport-ent-updater