Cluster Monitoring

Gravity Clusters come with a fully configured and customizable monitoring/alerting system by default. This system consists of the following components: Heapster, InfluxDB, Grafana and Kapacitor.

These components are automatically included into an Application Bundle built with tele build as a system dependency (see the source on GitHub).


Heapster monitors Kubernetes components and reports statistics and information to InfluxDB about nodes and pods.


InfluxDB is the main data store for current + future monitoring time series data. It provides the service influxdb.monitoring.svc.cluster.local.


Grafana is the dashboard system that provides visualization information on all the information stored in InfluxDB. It is exposed as the service grafana.monitoring.svc.cluster.local. Grafana credentials are generated during initial installation and placed into a secret grafana in monitoring namespace.


Kapacitor is the alerting system that streams data from InfluxDB and sends alerts as configured by the end user. It exposes the service kapacitor.monitoring.svc.cluster.local.

Grafana integration

The standard Grafana configuration includes two pre-configured dashboards providing machine- and pod-level overview of the installed cluster by default. The Grafana UI is integrated with Gravity control panel. To view dashboards once the cluster is up and running, navigate to the Cluster's Monitoring page.

By default, Grafana is running in anonymous read-only mode which allows anyone logged into Gravity to view existing dashboards (but not modify them or create new ones).

Pluggable dashboards

Your applications can use their own Grafana dashboards using ConfigMaps. A custom dashboard ConfigMap should be assigned a monitoring label with value dashboard and created in the monitoring namespace so it is recognized and loaded when installing the application:

apiVersion: v1
kind: ConfigMap
  name: mydashboard
  namespace: monitoring
    monitoring: dashboard
  mydashboard: |
    { ... dashboard JSON ... }

Dashboard ConfigMap may contain multiple keys with dashboards and key names are not relevant. The monitoring application source on GitHub has an example of a dashboard ConfigMap.

Since the embedded Grafana runs in read-only mode, a separate Grafana instance is required to create a custom dashboard, which can then be exported.

Metrics collection

All default metrics collected by Heapster go into the k8s database in InfluxDB. All other applications that collect metrics should submit them into the same database in order for proper retention policies to be enforced.

Retention policies

By default InfluxDB has 3 pre-configured retention policies:

The default retention policy is supposed to store high-precision metrics (for example, all default metrics collected by Heapster with 10s interval). The default policy is default for k8s database which means that metrics that do not specify retention policy explicitly go in there. The other two policies - medium and long are intended to store metric rollups and should not be used directly.

Durations for each of the retention policies can be configured through the Gravity Cluster control panel.


Metric rollups provide access to historical data for longer time period but at lower resolution.

The monitoring system allows for the configuration of two "types" of rollups for any collected metric.

Rollups are pre-configured for some of the metrics collected by default. Applications that collect their own metrics can configure their own rollups as well through ConfigMaps.

A custom rollup ConfigMap should be assigned a monitoring label with value rollup and created in the monitoring namespace for it to be recognized and loaded. Below is a sample ConfigMap:

apiVersion: v1
kind: ConfigMap
  name: myrollups
  namespace: monitoring
    monitoring: rollup
  rollups: |
        "retention": "medium",
        "measurement": "cpu/usage_rate",
        "name": "cpu/usage_rate/medium",
        "functions": [
            "function": "max",
            "field": "value",
            "alias": "value_max"
            "function": "mean",
            "field": "value",
            "alias": "value_mean"

Each rollup is a JSON object with the following fields:

List of supported functions

Aggregations: * count * distinct * integral * mean * median * mode * spread * stddev * sum

Selectors: * bottom * first * last * max * min * percentile * sample * top

Full documentation about functions can be found on InfluxDB website.

Kapacitor integration

Kapacitor provides alerting for default and user-defined alerts.


To configure Kapacitor to send email alerts, create resources of type smtp and alerttarget:

kind: smtp
version: v2
  name: smtp
  port: <smtp port> # 465 by default
  username: <username>
  password: <password>
kind: alerttarget
version: v2
  name: email-alerts
  email: [email protected] # Email address of the alert recipient
$ gravity resource create -f smtp.yaml

To create new alerts, use another resource of type alert:

kind: alert
version: v2
  name: my-formula
  formula: |
    Kapacitor formula

And introduce it with:

$ gravity resource create -f formula.yaml

To view SMTP configuration or alerts:

$ gravity resource get smtps smtp
$ gravity resource get alert my-formula

To remove an alert:

$ gravity resource rm alert my-formula

Builtin Alerts

Alerts (written in TICKscript) are automatically detected, loaded and enabled. They are read from the Kubernetes ConfigMap named kapacitor-alerts in monitoring namespace.

Following table shows the alerts Gravity ships with by default:

Component Alert Description
CPU High CPU usage Triggers a warning, when > 75% used, with > 90% used, triggers a critical error
Memory High memory usage Triggers a warning, when > 80% used, with > 90% used, triggers a critical error
Systemd Overall systemd health Triggers an error when systemd detects a failed service
Systemd Individual systemd unit health Triggers an error when a systemd unit is not loaded/active
Filesystem High disk space usage Triggers a warning, when > 80% used, with > 90% used, triggers a critical error
Filesystem High inode usage Triggers a warning, when > 90% used, with > 95% used, triggers a critical error
System Uptime Triggers a warning when a node's uptime is less than 5min
System Kernel parameters Triggers an error if a parameter is not set. See value matrix for details.
Etcd Etcd instance health Triggers an error when an Etcd master is down longer than 5min
Etcd Etcd latency check Triggers a warning, when follower <-> leader latency exceeds 500ms, then an error when it exceeds 1s over a period of 1min
Docker Docker daemon health Triggers an error when docker daemon is down
InfluxDB InfluxDB instance health Triggers an error when InfluxDB is inaccessible
Kubernetes Kubernetes node readiness Triggers an error when the node is not ready

Kapacitor will also trigger an email for each of the events listed above if SMTP resource has been configured (see configuration for details).

Gravity Enterprise

Gravity Enterprise enhances Gravity Community, the open-source Kubernetes packaging solution, to meet security and compliance requirements. It is trusted by some of the largest enterprises in software, finance, healthcare, security, telecom, government, and other industries.

Demo Gravity Enterprise

Gravity Community

Gravity Community is an upstream Kubernetes packaging solution that takes the drama out of on-premise deployments. Gravity Community is open-source software that anyone can download and install for free.

Download Gravity Community