Monitoring with Prometheus

Monitoring can be performed in SUSE Manager using Prometheus and Grafana. The packages for Prometheus and Grafana are shipped with SUSE Manager Client Tools, as well as packages for several Prometheus exporters. SUSE Manager Server and Proxy are now able to provide self-health metrics, or install and manage a limited number of Prometheus exporters on managed client systems.

Prometheus is a monitoring tool, originally built at SoundCloud, that is used to record real-time metrics in a time-series database. Unlike other monitoring systems, Prometheus collects metrics using HTTP pulls, allowing for higher performance and scalability. Prometheus is an open-source software project, mostly written in Go, and its source code is available at https://github.com/prometheus/.

Grafana is a tool for data visualization, monitoring and analysis. It is used to create dashboards with panels representing specific metrics over a set period of time. Grafana is commonly used together with Prometheus, but also supports other data sources such as ElasticSearch, MySQL, PostgreSQL, and Influx DB. For more information about Grafana, see: https://grafana.com/docs/.

A Prometheus package is included in the SUSE Manager Client Tools for SUSE Linux Enterprise 12 and SUSE Linux Enterprise 15.

Prometheus Metrics

Prometheus metrics are time series data, or timestamped values belonging to the same group or dimension. A metric is uniquely identified by its name and set of labels.

   metric name               labels                 timestamp     value
┌────────┴───────┐  ┌───────────┴───────────┐    ┌──────┴──────┐  ┌─┴─┐
http_requests_total{status="200", method="GET"}  @1557331801.111  42236

Each application or system being monitored must expose metrics in the format above, either through code instrumentation, or Prometheus exporters.

The different metric types are:

  • Counter - cumulative values. ex: number of errors

  • Gauge - can go up or down. ex: temperature

  • Histogram - count observations in buckets

  • Summary - similar to histogram, but provides totals (sum and count)

For more information about metric types, see: https://prometheus.io/docs/concepts/metric_types/

PromQL

Prometheus has its own query language called PromQL, which is a functional expression language. PromQL allows you to filter multi-dimensional time series data. It is used in all Prometheus interactions.

In PromQL, an expression can evaluate to one of three types:

  • Instant vector: a set of time series containing a single sample for each time series, all sharing the same timestamp

  • Range vector: a set of time series containing a range of data points over time for each time series

  • Scalar: a numeric floating point value

The core part of any PromQL query is the metric name, for example: http_requests_total. Labels can be used as optional selectors. This example returns the total number of HTTP requests that have status 200 and method GET:

http_requests_total{status="200", method="GET"}

For more information about PromQL, see the official Prometheus documentation: https://prometheus.io/docs/prometheus/latest/querying/basics/).

Exporters

Exporters are libraries which help in exporting existing metrics from third-party systems as Prometheus metrics. Exporters are useful whenever it is not feasible to instrument a given application or system with Prometheus metrics directly. Multiple exporters can run on a monitored host to export local metrics.

The Prometheus community provides a list of official exporters, and many others can be found as community contributions. For detailed information and an extensive list of exporters, see: https://prometheus.io/docs/instrumenting/exporters/.

With SUSE Manager 4, you can set up the Server and Proxy to expose Prometheus metrics to provide insights about SUSE Manager self-health. Metrics are available for these services:

  • Hardware and Operating System

  • Java Virtual Machines

  • Apache

  • Squid

  • PostgreSQL

  • SUSE Manager internals

The self-health metrics are made available by SUSE Manager Java application combined with Prometheus standalone exporters, running as systemd daemons.

SUSE Manager requires these packages to be installed on the Server and the Proxy. The packages are shipped with SUSE Manager Server and Proxy, but their respective systemd daemons are disabled by default.

These exporter packages are shipped with SUSE Manager Server:

These exporter packages are shipped with SUSE Manager Proxy:

Install and Configure Prometheus

Prometheus is installed from a package, and needs configuration before you can use it to gather metrics.

Installing Prometheus

Procedure: Installing Prometheus
  1. Install the golang-github-prometheus-prometheus package:

    zypper in golang-github-prometheus-prometheus
  2. Enable the Prometheus service:

    systemctl enable --now prometheus
  3. Confirm that the Prometheus interface is loading correctly. In your browser, navigate to the URL of the server where Prometheus is installed, on port 9090 (for example, http://example.com:9090).

Configuring Prometheus

Prometheus requires some configuration to collect metrics and set up alarms, or to display metrics graphically in Grafana. You can configure Prometheus in the static configuration file at /etc/prometheus/prometheus.yml. It is important to understand how this file is structured. For example:

yaml
- job_name: 'suse-manager-server'
  static_configs:
    - targets:
      - 'suse-manager.local:9100'  # Node exporter
      - 'suse-manager.local:9187'  # PostgreSQL exporter
      - 'suse-manager.local:5556'  # JMX exporter (Tomcat)
      - 'suse-manager.local:5557'  # JMX exporter (Taskomatic)
      - 'suse-manager.local:9800'  # Taskomatic
    - targets:
      - 'suse-manager.local:80'    # Message queue
      labels:
        __metrics_path__: /rhn/metrics

For more information about configuring Prometheus, see the official Prometheus documentation: https://prometheus.io/docs/prometheus/latest/configuration/configuration/

Monitoring Managed Systems

Prometheus metrics exporters can also be used on managed client systems. The packages are available from the SUSE Manager client tools channels, and can be enabled and configured directly on the SUSE Manager Web UI. Currently, two exporters are supported:

Installing and configuring exporters is done using a Salt formula.

Enable and Configure Monitoring

Procedure: Enabling Self Monitoring for SUSE Manager
  1. In the SUSE Manager Web UI, navigate to Admin  Manager Configuration  Monitoring.

  2. Click Enable services.

monitoring enable services
Procedure: Configuring Monitoring Formulas
  1. In the SUSE Manager Web UI, open the details page for the server, and navigate to the Formulas tab.

  2. Check the Monitoring checkbox to select all monitoring formulas, and click Save.

  3. Apply the highstate.

Procedure: Configuring the Exporters
  1. In the SUSE Manager Web UI, open the details page for the server, and navigate to the Formulas  Prometheus Exporters tab.

  2. Check the Enabled checkbox for both the Node Exporter, and the Postgres Exporter.

  3. In the Postgres Exporter section, in the Data Source Namer field, enter the path to your data source (for example postgresql://user:passwd@localhost:5432/database?sslmode=disable).

  4. Click Save Formula.

  5. Apply the highstate.

monitoring configure formula

Visualization with Grafana

The Grafana website contains dozens of dashboards uploaded by the community. For an example SUSE Manager dashboard to help you to get started, see https://grafana.com/dashboards/10277. For more information, see: https://grafana.com/dashboards

To use Grafana with SUSE Manager, you need to have enabled metrics in the SUSE Manager Web UI, and configured your Prometheus instance to collect those metrics.