percona/mongodb_exporter

MongoDB exporter

This is the new MongoDB exporter implementation that handles ALL metrics exposed by MongoDB monitoring commands.
This new implementation loops over all the fields exposed in diagnostic commands and tries to get data from them.

Currently, these metric sources are implemented:

• $collStats
• $indexStats
• getDiagnosticData
• replSetGetStatus
• serverStatus

Info on Percona MongoDB exporter versions

The old 0.1x.y version (ex master branch) has been moved to the release-0.1x branch.

A flag, --compatible-mode, which exposes metrics with 0.1x compatible metric
names has been implemented which simplifies migration from the old version to
the current version.

Build the exporter

The build process uses the dockerized version of goreleaser so you don't need to install Go.
Just run make release and the new binaries will be generated under the build directory.

├── build
│ ├── config.yaml
│ ├── mongodb_exporter_7c73946_checksums.txt
│ ├── mongodb_exporter-7c73946.darwin-amd64.tar.gz
│ ├── mongodb_exporter-7c73946.linux-amd64.tar.gz
│ ├── mongodb_exporter_darwin_amd64
│ │ └── mongodb_exporter <--- MacOS binary
│ └── mongodb_exporter_linux_amd64
│ └── mongodb_exporter <--- Linux binary

Running the exporter

If you built the exporter using the method mentioned in the previous section, the generated binaries are in mongodb_exporter_linux_amd64/mongodb_exporter or mongodb_exporter_darwin_amd64/mongodb_exporter

Docker

A docker image is available on the official percona repository.

Examples

# with podman
podman run -d -p 9216:9216 -p 17001:17001 percona/mongodb_exporter:0.20 --mongodb.uri=mongodb://127.0.0.1:17001

# with docker
docker run -d -p 9216:9216 -p 17001:17001 percona/mongodb_exporter:0.20 --mongodb.uri=mongodb://127.0.0.1:17001

Permissions

Connecting user should have sufficient rights to query needed stats:

      {
         "role":"clusterMonitor",
         "db":"admin"
      },
      {
         "role":"read",
         "db":"local"
      }

More info about roles in MongoDB documentation.

Example

mongodb_exporter_linux_amd64/mongodb_exporter --mongodb.uri=mongodb://127.0.0.1:17001

Enabling collstats metrics gathering

--mongodb.collstats-colls receives a list of databases and collections to monitor using collstats.
Usage example: --mongodb.collstats-colls=database1.collection1,database2.collection2

mongodb_exporter_linux_amd64/mongodb_exporter --mongodb.uri=mongodb://127.0.0.1:17001 --mongodb.collstats-colls=db1.c1,db2.c2

Enabling compatibility mode.

When compatibility mode is enabled by the --compatible-mode, the exporter will expose all new metrics with the new naming and labeling schema and at the same time will expose metrics in the version 1 compatible way.
For example, if compatibility mode is enabled, the metric mongodb_ss_wt_log_log_bytes_written (new format)

# HELP mongodb_ss_wt_log_log_bytes_written serverStatus.wiredTiger.log.
# TYPE mongodb_ss_wt_log_log_bytes_written untyped
mongodb_ss_wt_log_log_bytes_written 2.6208e+06

will be also exposed as mongodb_mongod_wiredtiger_log_bytes_total with the unwritten label.

HELP mongodb_mongod_wiredtiger_log_bytes_total mongodb_mongod_wiredtiger_log_bytes_total
# TYPE mongodb_mongod_wiredtiger_log_bytes_total untyped
mongodb_mongod_wiredtiger_log_bytes_total{type="unwritten"} 2.6208e+06

Cluster role labels

The exporter sets some topology labels in all metrics.
The labels are:

• cl_role: Cluster role according to this table:

• cl_id: Cluster ID
• rs_nm: Replicaset name
• rs_state: Replicaset state is an integer from getDiagnosticData() -> replSetGetStatus.myState.
  Check the official documentation for details on replicaset status values.

Usage Reference

See the Reference Guide for details on using the exporter.

Bug Reports / Feature PR

Refer to the Contribution Guide.

MongoDB Exporter Helm Chart

Helm chart to install MySQL

If your MongoDB is not up and ready yet, you can start the MongoDB cluster using Helm:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm upgrade --install my-release bitnami/mongodb --set architecture=replicaset --set metrics.enabled=true --set metrics.extraFlags="--compatible-mode"

With these commands, MongoDB will be up and running with a sidecar container to expose Prometheus metrics.

Please note:  “--compatible-mode” is enabled explicitly to expose all kinds of metrics (new and old format), so that most of the community dashboards can be used without alterations.

In the case of an existing MongoDB or a MongoDB running outside of Kubernetes, we need to deploy an explicit exporter to get the metrics. For this, follow the steps below:

Installing MongoDB Exporter

The MongoDB exporter can be deployed in Kubernetes using the Helm chart. The Helm chart used for deployment is from the Prometheus community and can be found here. To deploy this Helm chart, users can either follow the steps in the above link or refer to the ones outlined below:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install [RELEASE_NAME] prometheus-community/prometheus-mongodb-exporter

Some of the common parameters that should be changed in the values file include: 

mongodb.uri: <[mongodb[+srv]://][user:pass@]host1[:port1][,host2[:port2],...][/database][?options]>

Ex:

mongodb.uri:  “mongodb://root:password@mongodb.default.svc.cluster.local:27017/admin?authSource=admin”

In case the user wants to pass the credentials as secrets, they can create a secret and pass the secret name in the following value file.

existingSecret.name: <Secret name>

Additional parameters can be changed based on the individual needs - such as enabling and disabling collectors, parameters, and so on. All these parameters can be tuned via values.yaml file here.

In addition to the native way of setting up Prometheus monitoring, a service monitor can be deployed (if the Prometheus operator is being used) to scrap the data from MongoDB. Prometheus then scraps the data from the service monitor. With this approach, multiple MongoDB can be scrapped without altering the Prometheus configuration. Every MongoDB exporter comes with its own service monitor.

In the above-mentioned chart, a service monitor can be deployed by turning it on from the values.yaml file here. By default, it is set to true.

serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s
  namespace:
  additionalLabels: {}
  targetLabels: []
  metricRelabelings: []

Another way of scraping metrics while having the pod discovery enabled in Prometheus is by updating the annotation section here with the following:

 ​​podAnnotations:
    prometheus.io/path: /metrics
    prometheus.io/scrape: "true"

Here is a sample values file:

mongodb:
  uri: ""

# Name of an externally managed secret (in the same namespace) containing the connection uri as key `mongodb-uri`.
# If this is provided, the value mongodb.uri is ignored.
existingSecret:
  name: ""
  key: "mongodb-uri"

nameOverride: ""

nodeSelector: {}

podAnnotations: {}
#  prometheus.io/scrape: "true"
#  prometheus.io/port: "metrics"

port: "9216"

priorityClassName: ""

service:
  labels: {}
  annotations: {}
  port: 9216
  type: ClusterIP

serviceAccount:
  create: true
  # If create is true and name is not set, then a name is generated using the
  # fullname template.
  name:

serviceMonitor:
  enabled: true
  interval: 30s
  scrapeTimeout: 10s
  namespace:
  additionalLabels: {}
  targetLabels: []
  metricRelabelings: []

This concludes our review of the exporter for MongoDB. Please feel free to reach out to us via support@nexclipper.io in case you have any questions. As always, stay tuned for more exporter reviews and tips coming soon!

MongoDB Exporter Alerts

After digging into all the valuable metrics, this section explains in detail how we can get critical alerts.

PromQL is a query language for the Prometheus monitoring system. It is designed for building powerful yet simple queries for graphs, alerts, or derived time series (aka recording rules). PromQL is designed from scratch and has zero common grounds with other query languages used in time series databases, such as SQL in TimescaleDB, InfluxQL, or Flux. More details can be found here.

Prometheus comes with a built-in Alert Manager that is responsible for sending alerts (could be email, Slack, or any other supported channel) when any of the trigger conditions is met. Alerting rules allow users to define alerts based on Prometheus query expressions. They are defined based on the available metrics scraped by the exporter. Click here for a good source for community-defined alerts.

A general alert looks as follows:

- alert:(Alert Name)
expr: (Metric exported from exporter) >/</==/<=/=> (Value)
for: (wait for a certain duration between first encountering a new expression output vector element and counting an alert as firing for this element)
labels: (allows specifying a set of additional labels to be attached to the alert)
annotation: (specifies a set of informational labels that can be used to store longer additional information)

Some of the recommended MongoDB alerts are:

1. Alert - MongoDB is Down

- alert: MongodbDown
  expr: mongodb_up == 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: MongoDB Down (instance {{ $labels.instance }})
    description: "MongoDB instance is down\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

2. Alert - MongoDB has too many connections

- alert: MongodbTooManyConnections
  expr: avg by(instance) (rate(mongodb_connections{state="current"}[1m])) / avg by(instance) (sum (mongodb_connections) by (instance)) * 100 > 80
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: MongoDB too many connections (instance {{ $labels.instance }})
    description: "Too many connections (> 80%)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

3.  Alert - MongoDB replication lag

• Prometheus community:

 - alert: MongodbReplicationLag
   expr: mongodb_mongod_replset_member_optime_date{state="PRIMARY"}   - ON (set) mongodb_mongod_replset_member_optime_date{state="SECONDARY"} > 10
   for: 0m
   labels:
     severity: critical
   annotations:
     summary: MongoDB replication lag (instance {{ $labels.instance }})
     description: "Mongodb replication lag is more than 10s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

• Bitnami:

  - alert: MongodbReplicationLag
    expr: avg(mongodb_replset_member_optime_date{state="PRIMARY"}) - avg(mongodb_replset_member_optime_date{state="SECONDARY"}) > 10
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: MongoDB replication lag (instance {{ $labels.instance }})
      description: "Mongodb replication lag is more than 10s\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

4.   Alert -  MongoDB replication status (for Prometheus community exporter)

  - alert: MongodbReplicationStatus8
    expr: mongodb_mongod_replset_member_state == 8
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: MongoDB replication Status 8 (instance {{ $labels.instance }})
      description: "MongoDB Replication set member as seen from another member of the set, is unreachable\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

5.  Alert - High memory usage

 - alert: MongodbVirtualMemoryUsage
   expr: (sum(mongodb_memory{type="virtual"}) BY (instance) / sum(mongodb_memory{type="mapped"}) BY (instance)) > 3
   for: 2m
   labels:
     severity: warning
   annotations:
     summary: MongoDB virtual memory usage (instance {{ $labels.instance }})
     description: "High memory usage\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Additionally, here are some other useful alerts:

1. Alert - MongoDB curser time out - happens when too many operations are happening on MongoDB

• With Prometheus community exporter:

  - alert: MongodbCursorsTimeouts
    expr: increase(mongodb_metrics_cursor_timed_out_total[1m]) > 100
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: MongoDB cursors timeouts (instance {{ $labels.instance }})
      description: "Too many cursors are timing out\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

• With Bitnami:

  - alert: MongodbCursorsTimeouts
    expr: increase(mongodb_mongod_metrics_cursor_timed_out_total[1m]) > 100
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: MongoDB cursors timeouts (instance {{ $labels.instance }})
      description: "Too many cursors are timing out\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

2. Alert - Too many cursor open for clients

• For the Prometheus community exporter:

  - alert: MongodbNumberCursorsOpen
    expr: mongodb_mongod_metrics_cursor_open{state="total"} > 10 * 1000
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: MongoDB number cursors open (instance {{ $labels.instance }})
      description: "Too many cursors opened by MongoDB for clients (> 10k)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

• For Bitnami:

  - alert: MongodbNumberCursorsOpen
    expr: mongodb_metrics_cursor_open{state="total_open"} > 10000
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: MongoDB number cursors open (instance {{ $labels.instance }})
      description: "Too many cursors opened by MongoDB for clients (> 10k)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

MongoDB Exporter Grafana

Graphs are easier to understand and more user-friendly than a row of numbers. For this purpose, users can plot their time series data in visualized format using Grafana.

Grafana is an open-source dashboarding tool used for visualizing metrics with the help of customizable and illustrative charts and graphs. It connects very well with Prometheus and makes monitoring easy and informative. Dashboards in Grafana are made up of panels, with each panel running a PromQL query to fetch metrics from Prometheus.
Grafana supports community-driven graphs for most of the widely used software, which can be directly imported to the Grafana Community.

What is a Panel?

Panels are the most basic component of a dashboard and can display information in various ways, such as gauge, text, bar chart, graph, and so on. They provide information in a very interactive way. Users can view every panel separately and check the value of metrics within a specific time range. 
The values on the panel are queried using PromQL, which is Prometheus Query Language. PromQL is a simple query language used to query metrics within Prometheus. It enables users to query data, aggregate and apply arithmetic functions to the metrics, and then further visualize them on panels.

Here are some examples of panels: