Installation

Ops Center-driven Installation

An Ops Center can be used for application deployments. In order to deploy applications from an Ops Center, Application Bundles are published to an Ops Center. Once published, they become available for deployment. An Application can be deployed either directly from an Ops Center or via a one-time installation link. Once the cluster is up and running, the installer will establish a remote access channel for maintenance:

Ops Center Install

NOTE

The end users can close the remote channel and disconnect their Application from the Ops Center.

Standalone Offline UI Installation

Standalone installation allows users to install into air-gapped (offline) server clusters using a self-contained installer tarball.

  • The tarball can be generated on the Ops Center, by using the Download link.
  • Alternatively, the installer can be fetched from an Ops Center using tele pull, see Publishing Applications section for details.
  • Installer tarball can be built with the tele build command, see Packaging Applications for details.

For example, Application Bundle, my-app-installer.tar will contain everything required to install the Application my-app.

To install using the graphical wizard, a Linux desktop with a browser is required and the target servers need to be reachable on port 3012. The node running wizard should have port 61009 reachable to other servers.

Unpacking the tarball will produce the following contents:

$ tar -xf my-app-installer.tar
$ ls -lh
-rwxr--r-- 1 user staff 21K  Oct 24 12:01 app.yaml
-rwxr--r-- 1 user staff 56M  Oct 24 12:01 gravity
-rwxr--r-- 1 user staff 256K Oct 24 12:01 gravity.db
-rwxr--r-- 1 user staff 679  Oct 24 12:01 install
-rwxr--r-- 1 user staff 170  Oct 24 12:01 packages
-rw-r--r-- 1 user staff 1.1K Oct 24 12:01 README
-rwxr--r-- 1 user staff 170  Oct 24 12:01 upgrade
-rwxr--r-- 1 user staff 170  Oct 24 12:01 upload

The installation wizard is launched by typing ./install and will guide the end user through the installation process.

Telekube Offline Installer

Standalone Offline CLI Installation

Instead of running a graphical installer, an Application Bundle can be installed through a CLI which is useful for integrations with configuration management scripts or other types of infrastructure automation tools. Sometimes this method is called "unattended installation".

For this to work, the information needed to complete the installation has to be supplied via the command line flags to the installer.

Let's see how to install a 3-node cluster:

  1. Copy the Application Bundle onto all nodes.
  2. Execute ./gravity install on the first node.
  3. Execute ./gravity join on two other nodes.

Below is a sample ./gravity install command for the first node:

node-1$ sudo ./gravity install --advertise-addr=172.28.128.3 --token=XXX --flavor="three"

Note the use of flavor which, in this case, describes a configuration for 3 nodes ("three" being the name of the flavor from the Application Manifest).

This will initiate the process of setting up a new cluster for the Application.

Below are corresponding ./gravity join commands for the remaining nodes (node-2 and node-3):

node-2$ sudo ./gravity join 172.28.128.3 --advertise-addr=172.28.128.4 --token=XXX --role="database"
node-3$ sudo ./gravity join 172.28.128.3 --advertise-addr=172.28.128.5 --token=XXX --role="worker"

This instructs the nodes to join a cluster initiated by gravity install on the node 172.28.128.3. Note, that nodes have also been assigned the desired roles (as defined in the Application Manifest).

The install command accepts the following arguments:

Flag Description
--token Secure token which prevents rogue nodes from joining the cluster during installation. Carefully pick a hard-to-guess value.
--advertise-addr IP address this node should be visible as. This setting is needed to correctly configure Telekube on every node.
--role (Optional) Application role of the node.
--cluster (Optional) Name of the cluster. Auto-generated if not set.
--cloud-provider (Optional) Enable cloud provider integration: generic (no cloud provider integration), aws or gce. Autodetected if not set.
--flavor (Optional) Application flavor. See Application Manifest for details.
--config (Optional) File with Kubernetes resources to create in the cluster during installation.
--pod-network-cidr (Optional) CIDR range Kubernetes will be allocating node subnets and pod IPs from. Must be a minimum of /16 so Kubernetes is able to allocate /24 to each node. Defaults to 10.244.0.0/16.
--service-cidr (Optional) CIDR range Kubernetes will be allocating service IPs from. Defaults to 10.100.0.0/16.
--wizard (Optional) Start the installation wizard.
--state-dir (Optional) Directory where all Telekube system data will be kept on this node. Defaults to /var/lib/gravity.
--service-uid (Optional) Service user ID (numeric). See Service User for details. A user named planet is created automatically if unspecified.
--service-gid (Optional) Service group ID (numeric). See Service User for details. A group named planet is created automatically if unspecified.
--dns-zone (Optional) Specify an upstream server for the given DNS zone within the cluster. Accepts <zone>/<nameserver> format where <nameserver> can be either <ip> or <ip>:<port>. Can be specified multiple times.

The join command accepts the following arguments:

Flag Description
--token Secure token which prevents rogue nodes from joining the cluster during installation. Carefully pick a hard-to-guess value.
--advertise-addr IP address this node should be visible as. This setting is needed to correctly configure Telekube on every node.
--role (Optional) Application role of the node.
--cloud-provider (Optional) Cloud provider integration, generic or aws. Autodetected if not set.
--mounts (Optional) Comma-separated list of mount points as :.
--state-dir (Optional) Directory where all Telekube system data will be kept on this node. Defaults to /var/lib/gravity.
--service-uid (Optional) Service user ID (numeric). See Service User for details. A user named planet is created automatically if unspecified.
--service-gid (Optional) Service group ID (numeric). See Service User for details. A group named planet is created automatically if unspecified.

NOTE

--advertise-addr must also be set for every node, and the same value for --token must be used.

NOTE

With no role specified, the installer uses the first role defined in the Application Manifest.

The result of running these commands will be a functional and self-contained Kubernetes cluster!

You can learn more in the Packaging and Deployment section of the documentation.

Troubleshooting Installs

The installation process is implemented as a state machine split into multiple steps (phases). Every time a step fails, the install pauses and allows one to inspect and correct the cause of the failure.

If the installation has failed, the installer will print a warning and pause:

root$ ./gravity install
Tue Apr 10 13:44:07 UTC Starting installer
Tue Apr 10 13:44:09 UTC Preparing for installation
Tue Apr 10 13:44:32 UTC Installing application my-app:1.0.0-rc.1
Tue Apr 10 13:44:32 UTC Starting non-interactive install
Tue Apr 10 13:44:32 UTC Bootstrapping local state
Tue Apr 10 13:44:33 UTC All agents have connected!
Tue Apr 10 13:44:33 UTC Starting the installation
Tue Apr 10 13:44:34 UTC Operation has been created
Tue Apr 10 13:44:35 UTC Execute preflight checks
Tue Apr 10 13:44:37 UTC Operation has failed
Tue Apr 10 13:44:37 UTC Installation failed in 4.985481556s, check ./telekube-install.log
---
Installer process will keep running so you can inspect the operation plan using
`gravity plan` command, see what failed and continue plan execution manually
using `gravity install --phase=<phase-id>` command after fixing the problem.
Once no longer needed, this process can be shutdown using Ctrl-C.

To inspect installer's progress, use the plan command:

root$ ./gravity plan
Phase                  Description                                                               State         Requires                  Updated
-----                  -----------                                                               -----         --------                  -------
⚠ checks               Execute preflight checks                                                  Failed        -                         Tue Apr 10 13:44 UTC
* configure            Configure packages for all nodes                                          Unstarted     -                         -
* bootstrap            Bootstrap all nodes                                                       Unstarted     -                         -
  * node-1             Bootstrap master node node-1                                              Unstarted     -                         -
* pull                 Pull configured packages                                                  Unstarted     /configure,/bootstrap     -
  * node-1             Pull packages on master node node-1                                       Unstarted     /configure,/bootstrap     -
* masters              Install system software on master nodes                                   Unstarted     /pull                     -
  * node-1             Install system software on master node node-1                             Unstarted     /pull/node-1
...

Phase Execute preflight checks (/checks) failed.
Error:
server("node-1", 192.168.121.23) failed checks:
    ⚠ fs.may_detach_mounts should be set to 1 or pods may get stuck in the Terminating state, see https://www.gravitational.com/docs/faq/#kubernetes-pods-stuck-in-terminating-state

After fixing the error (i.e. enabling the kernel parameter in this example), resume the installation:

root$ sysctl -w fs.may_detach_mounts=1
root$ ./gravity install --resume
Tue Apr 10 13:55:26 UTC Executing "/checks" locally
Tue Apr 10 13:55:26 UTC Running pre-flight checks
Tue Apr 10 13:55:28 UTC Executing "/configure" locally
Tue Apr 10 13:55:28 UTC Configuring cluster packages
Tue Apr 10 13:55:32 UTC Executing "/bootstrap/node-1" locally
Tue Apr 10 13:55:32 UTC Configuring system directories
Tue Apr 10 13:55:35 UTC Configuring application-specific volumes
Tue Apr 10 13:55:36 UTC Executing "/pull/node-1" locally
Tue Apr 10 13:55:36 UTC Pulling user application
Tue Apr 10 13:55:46 UTC Still pulling user application (10 seconds elapsed)
...
Tue Apr 10 14:01:07 UTC Executing "/app/my-app" locally
Tue Apr 10 14:01:08 UTC Executing "/election" locally
Tue Apr 10 14:01:08 UTC Enable leader elections
Tue Apr 10 14:01:09 UTC Executing install phase "/" finished in 5 minutes

The following CLI flags are useful to manage the install operation:

Flag Description
--phase Specifies the name of the step to execute. Use gravity plan to display the list of all steps.
--force Force execution of the step even it is already in-progress.
--resume Resume operation after the failure. The operation is resumed from the step that failed last.
--manual Launch operation in manual mode.

AWS Installer

A standalone installer can be used to create a new AWS cluster from a deployable tarball using automatic provisioner. It works by uploading the installer tarball on a AWS EC2 instance and executing the gravity install command.

Restrictions & Requirements

This install flow is subject to a few restrictions and requirements.

  • The installer must be launched from a EC2 instance.
  • The EC2 instance that runs the installer will not be part of the cluster.
  • The EC2 instance that runs the installer must have a public IP attached, or be otherwise reachable by the nodes of the cluster that is being installed.
  • The EC2 instance that runs the installer must have ports 32009 and 3024 open to the nodes of the cluster that is being installed.
  • The EC2 instance that runs the installer must be able to accommodate a single-node Telekube cluster of its own so all requirements and recommendations from System Requirements apply to it.

AWS Credentials & IAM Policy

In addition to the requirements listed above, the machine running the installer should have proper AWS credentials. The credentials can be configured by creating a AWS credentials file as described in Configuring the AWS CLI. If credentials are not found, the installer will attempt to retrieve the instance's IAM credentials using AWS metadata API. In this case the instance should have a proper IAM policy assigned to it.

Click here to view an example IAM policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:*",
                "ec2:*",
                "elasticloadbalancing:*",
                "iam:AddRoleToInstanceProfile",
                "iam:CreateInstanceProfile",
                "iam:CreateRole",
                "iam:DeleteInstanceProfile",
                "iam:DeleteRole",
                "iam:DeleteRolePolicy",
                "iam:GetInstanceProfile",
                "iam:GetRole",
                "iam:GetRolePolicy",
                "iam:ListInstanceProfiles",
                "iam:ListInstanceProfilesForRole",
                "iam:ListRoles",
                "iam:PassRole",
                "iam:PutRolePolicy",
                "iam:RemoveRoleFromInstanceProfile",
                "kms:DescribeKey",
                "kms:ListAliases",
                "kms:ListKeys",
                "s3:*",
                "sqs:ChangeMessageVisibility",
                "sqs:ChangeMessageVisibilityBatch",
                "sqs:CreateQueue",
                "sqs:DeleteMessage",
                "sqs:DeleteMessageBatch",
                "sqs:DeleteQueue",
                "sqs:GetQueueAttributes",
                "sqs:GetQueueUrl",
                "sqs:ListDeadLetterSourceQueues",
                "sqs:ListQueueTags",
                "sqs:ListQueues",
                "sqs:PurgeQueue",
                "sqs:ReceiveMessage",
                "sqs:SendMessage",
                "sqs:SendMessageBatch",
                "sqs:SetQueueAttributes",
                "sqs:TagQueue",
                "sqs:UntagQueue",
                "ssm:DeleteParameter",
                "ssm:DeleteParameters",
                "ssm:DescribeParameters",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:ListTagsForResource",
                "ssm:PutParameter"
            ],
            "Resource": "*"
        }
    ]
}

Note

The exact list of required permissions may depend on the provisioner you're using. The IAM policy shown above is an example for gravitational/provisioner.

Installation

Once you have an instance that satisfies the above requirements, it can be used to install clusters on AWS using the flow described below.

  • Build the application installer, e.g. using tele build.

Note

Note that the application must include provisioning hooks as described in Customized Cluster Provisioning. See gravitational/provisioner repository on Github for an example of a Terraform-based provisioner.

  • Transfer the installer to the EC2 instance and unpack.
  • Create a cluster resource spec to describe the desired cluster profile:
# cluster.yaml
kind: cluster
version: v2
metadata:
  # cluster name
  name: example.com
spec:
  # cloud provider, must be "aws"
  provider: aws
  aws:
    # AWS region to install to
    region: us-east-2
    # AWS key pair name to use
    keyName: ops
    # (optional) AWS VPC to install to
    vpc: ""
  # nodes to provision
  nodes:
    # profile names must match those from the app manifest
    - profile: node
      count: 1
      instanceType: m4.xlarge
  # (optional) additional Kubernetes resources to create in cluster
  resources: |
    ...
  # (optional) license string if the app requires it
  license: |
    ...
  • Launch installation by running the following command from the installer tarball:
$ sudo ./gravity install --cluster-spec=cluster.yaml

Keep in mind that, as was mentioned in the requirements section above, by default the provisioned nodes will be connecting to the installer instance using its public IP which will be determined automatically by querying AWS metadata API. To override this behavior, you can explicitly set the advertise address using --ops-advertise-addr flag. Note that the port in the provided address must be 32009.

$ sudo ./gravity install --cluster-spec=cluster.yaml --ops-advertise-addr=172.31.41.70:32009

Installing on Google Compute Engine

Note

GCE cloud provider integration is supported starting from Telekube version 5.1.0-alpha.1.

Before installation make sure that GCE instances used for installation satisfy all of Telekube system requirements. In addition to these generic requirements GCE nodes also must be configured in the following way to ensure proper cloud provider integration:

  • Network interface must have IP forwarding turned on. It is required for the overlay network to work properly.
  • Instances must be assigned a network tag matching the name of the cluster. It is required to ensure that created load balancers discover proper instances.
  • Cloud API access scopes must include RW permissions for Compute Engine.

Once the nodes have been properly configured, copy the installer tarball and launch installation as described above:

node1$ sudo ./gravity install --advertise-addr=<addr> --token=<token> --cluster=<cluster> --cloud-provider=gce
node2$ sudo ./gravity join <installer-addr> --advertise-addr=<addr> --token=<token> --cloud-provider=gce

Note that the --cloud-provider flag is optional and, if unspecified, will be auto-detected if install/join process is running on a GCE instance.