Redis Exporter

Review
Exporters
Helm Charts
Rules
Dashboards
Discussion

This review discusses the Redis exporter, necessary for exposing and monitoring metrics from Redis, an in-memory data structure store used as a database, cache, streaming engine, and message broker.

About Redis

Redis stands for Remote Dictionary Server and is an in-memory data structure store used as a database, cache, streaming engine, and message broker. It provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, geospatial indexes, and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster.

A Redis exporter is required to monitor and expose Redis' metrics. It queries Redis, scraps the data, and exposes the metrics to a Kubernetes service endpoint that can further be scrapped by Prometheus to ingest the time series data. For monitoring Redis, we use an external Prometheus exporter, which is maintained by the Prometheus Community. On deployment, this exporter scraps sizable metrics from Redis and helps users get crucial information that is difficult to get from Redis directly and continuously. 

For this setup, we are using bitnami redis Helm charts to start the Redis server/cluster. 

How do you set up an exporter for Prometheus?

With the latest version of Prometheus (2.33 as of February 2022), these are the ways to set up a Prometheus exporter: 

Method 1 - Basic

Supported by Prometheus since the beginning
To set up an exporter in the native way a Prometheus config needs to be updated to add the target.
A sample configuration:

# scrape_config job scrape_configs: - job_name: redis scrape_interval: 45s scrape_timeout: 30s metrics_path: "/metrics" static_configs: - targets: - <redis exporter endpoint>
Code language: PHP (php)

Sample config for multiple Redis hosts:

## config for the multiple Redis targets that the exporter will scrape - job_name: 'redis_exporter_targets' static_configs: - targets: - redis://first-redis-host:6379 - redis://second-redis-host:6379 - <and so on> metrics_path: /scrape relabel_configs: - source_labels: [__address__] target_label: __param_target - source_labels: [__param_target] target_label: instance - target_label: __address__ replacement: <REDIS-EXPORTER-HOSTNAME>:9121
Code language: PHP (php)
Method 2 - Service Discovery

This method is applicable for Kubernetes deployment only.
A default scrap config can be added to the prometheus.yaml file and an annotation can be added to the exporter service. With this, Prometheus will automatically start scrapping the data from the services with the mentioned path.

Prometheus.yaml

- job_name: kubernetes-services scrape_interval: 15s scrape_timeout: 10s kubernetes_sd_configs: - role: service relabel_configs: # Example relabel to scrape only endpoints that have # prometheus.io/scrape: "true" annotation. - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true # prometheus.io/path: "/scrape/path" annotation. - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) # prometheus.io/port: "80" annotation. - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: (.+)(?::\d+);(\d+) replacement: $1:$2
Code language: PHP (php)

Exporter service annotations:

annotations: prometheus.io/path: /metrics prometheus.io/scrape: "true"
Code language: PHP (php)
Method 3 - Prometheus Operator

Setting up a service monitor
The Prometheus operator supports an automated way of scraping data from the exporters by setting up a service monitor Kubernetes object. For reference, a sample service monitor for Redis can be found here.
These are the necessary steps:

Step 1

Add/update Prometheus operator’s selectors. By default, the Prometheus operator comes with empty selectors which will select every service monitor available in the cluster for scrapping the data.

To check your Prometheus configuration:

Kubectl get prometheus -n <namespace> -o yaml
Code language: HTML, XML (xml)

A sample output will look like this.

ruleNamespaceSelector: {} ruleSelector: matchLabels: app: kube-prometheus-stack release: kps scrapeInterval: 1m scrapeTimeout: 10s securityContext: fsGroup: 2000 runAsGroup: 2000 runAsNonRoot: true runAsUser: 1000 serviceAccountName: kps-kube-prometheus-stack-prometheus serviceMonitorNamespaceSelector: {} serviceMonitorSelector: matchLabels: release: kps
Code language: CSS (css)

Here you can see that this Prometheus configuration is selecting all the service monitors with the label release = kps

So with this, if you are modifying the default Prometheus operator configuration for service monitor scrapping, make sure you use the right labels in your service monitor as well.

Step 2

Add a service monitor and make sure it has a matching label and namespace for the Prometheus service monitor selectors (serviceMonitorNamespaceSelector & serviceMonitorSelector).

Sample configuration:

apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: annotations: meta.helm.sh/release-name: redis-exporter meta.helm.sh/release-namespace: monitor labels: app: prometheus-redis-exporter app.kubernetes.io/managed-by: Helm chart: prometheus-redis-exporter-1.1.0 heritage: Helm release: kps name: redis-exporter-prometheus-redis-exporter namespace: monitor spec: endpoints: - interval: 15s port: redis-exporter selector: matchLabels: app: prometheus-redis-exporter release: redis-exporter

As you can see, a matching label on the service monitor release = kps is used that is specified in the Prometheus operator scrapping configuration.

Metrics

The following metrics are handpicked and will provide insights into Redis operations.

  1. Server is up
    As the name suggests, this metric will expose the state of the Redis process and whether it is up or down.
    ➡ The key of the exporter metric is “redis_up”
    ➡ The value of the metric is a boolean -  1 or 0 which symbolizes if Redis is up or down respectively (1 for yes, 0 for no) 
  1. Redis used memory
    Since Redis is an in-memory database, it is important to monitor the memory as full memory may cause data loss based on the maxmemory-policy configured.
    ➡ The key of the exporter metrics is “redis_memory_used_bytes”
    ➡ You can calculate the percentage based on “redis_total_system_memory_bytes” or “redis_memory_max_bytes
    ➡ The exporter must be started with “--include-system-metrics“ flag or the “REDIS_EXPORTER_INCL_SYSTEM_METRICS=true“ environment variable to show the “redis_total_system_memory_bytes” metric
    ➡ “redis_memory_max_bytes“ is 0 by default, to get the value you need to limit the memory used by Redis by running this command “CONFIG SET maxmemory <value>”
  1.  Too many connections
    Redis has a specific maximum number of clients configured and if the number of connections exceeds this value it rejects new connections.
    ➡ The metric “ redis_connected_clients” gives the total connections on Redis
    ➡ The number should be calculated based on the max allowed clients from the metric “redis_config_maxclients”
    ➡ Rejected clients can be monitored with "redis_rejected_connections_total"
  1. Redis rejecting connections
    This means Redis has reached a value at which it rejects accepting new connections. 
    ➡ The metric “redis_rejected_connections_total” gives the total rejected connections by Redis
    ➡ The value of this is a number that tells the average rate of the rejected connection based on the specified time
  1. Redis out of system memory
    This means Redis is going out of the host system memory it is hosted upon.
    ➡ “redis_memory_used_bytes” and “redis_memory_max_bytes” metrics provide the corresponding memory of Redis and the host
    ➡ The value of these metrics is returned as a number of bytes that can be calculated to process the data

Additionally, below are some of the metrics that are important for the Redis cluster:

  1. Redis missing master
    This means the master node is missing from the Redis cluster.
    ➡ “redis_instance_info{role="master"}”  is the key to get details about the master
    ➡ The value of this metric is a number that should be greater than 1
  1. Redis disconnected slaves
    This means slaves are not connected to the master.
    ➡ The metrics “redis_connected_slaves” provides the value of connected slaves, so to get the value of disconnected we need to subtract the provided value from the total available slaves
    ➡ The value of this is a number
  1. Redis replication is broken
    This means that the replication is broken between master and replica.
    ➡ The key “redis_connected_slaves” gives the value of connected slaves - if we take the delta of a minute it will tell us the replication state
    ➡ The value of this is a number
  • oliver006/redis_exporter
  • oliver006/redis_exporter

    Prometheus Redis Metrics Exporter

    Build Status
    Coverage Status
    codecov
    docker_pulls
    Stand With Ukraine

    Prometheus exporter for Redis metrics.\
    Supports Redis 2.x, 3.x, 4.x, 5.x, 6.x, and 7.x

    Ukraine is currently suffering from Russian aggression, please consider supporting Ukraine with a donation.

    Stand With Ukraine

    Building and running the exporter

    Build and run locally

    git clone https://github.com/oliver006/redis_exporter.git
    cd redis_exporter
    go build .
    ./redis_exporter --version

    Pre-build binaries

    For pre-built binaries please take a look at the releases.

    Basic Prometheus Configuration

    Add a block to the scrape_configs of your prometheus.yml config file:

    scrape_configs:
      - job_name: redis_exporter
        static_configs:
        - targets: ['<<REDIS-EXPORTER-HOSTNAME>>:9121']

    and adjust the host name accordingly.

    Kubernetes SD configurations

    To have instances in the drop-down as human readable names rather than IPs, it is suggested to use instance relabelling.

    For example, if the metrics are being scraped via the pod role, one could add:

              - source_labels: [__meta_kubernetes_pod_name]
                action: replace
                target_label: instance
                regex: (.*redis.*)

    as a relabel config to the corresponding scrape config. As per the regex value, only pods with "redis" in their name will be relabelled as such.

    Similar approaches can be taken with other role types depending on how scrape targets are retrieved.

    Prometheus Configuration to Scrape Multiple Redis Hosts

    Run the exporter with the command line flag --redis.addr= so it won't try to access the local instance every time the /metrics endpoint is scraped. Using below config instead of the /metric endpoint the /scrape endpoint will be used by prometheus. As an example the first target will be queried with this web request:
    http://exporterhost:9121/scrape?target=first-redis-host:6379

    scrape_configs:
      ## config for the multiple Redis targets that the exporter will scrape
      - job_name: 'redis_exporter_targets'
        static_configs:
          - targets:
            - redis://first-redis-host:6379
            - redis://second-redis-host:6379
            - redis://second-redis-host:6380
            - redis://second-redis-host:6381
        metrics_path: /scrape
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: <<REDIS-EXPORTER-HOSTNAME>>:9121
    
      ## config for scraping the exporter itself
      - job_name: 'redis_exporter'
        static_configs:
          - targets:
            - <<REDIS-EXPORTER-HOSTNAME>>:9121

    The Redis instances are listed under targets, the Redis exporter hostname is configured via the last relabel_config rule.\
    If authentication is needed for the Redis instances then you can set the password via the --redis.password command line option of
    the exporter (this means you can currently only use one password across the instances you try to scrape this way. Use several
    exporters if this is a problem). \
    You can also use a json file to supply multiple targets by using file_sd_configs like so:

    scrape_configs:
      - job_name: 'redis_exporter_targets'
        file_sd_configs:
          - files:
            - targets-redis-instances.json
        metrics_path: /scrape
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: <<REDIS-EXPORTER-HOSTNAME>>:9121
    
      ## config for scraping the exporter itself
      - job_name: 'redis_exporter'
        static_configs:
          - targets:
            - <<REDIS-EXPORTER-HOSTNAME>>:9121

    The targets-redis-instances.json should look something like this:

    [
      {
        "targets": [ "redis://redis-host-01:6379", "redis://redis-host-02:6379"],
        "labels": { }
      }
    ]

    Prometheus uses file watches and all changes to the json file are applied immediately.

    Command line flags

    NameEnvironment Variable NameDescription
    redis.addrREDIS_ADDRAddress of the Redis instance, defaults to redis://localhost:6379.
    redis.userREDIS_USERUser name to use for authentication (Redis ACL for Redis 6.0 and newer).
    redis.passwordREDIS_PASSWORDPassword of the Redis instance, defaults to "" (no password).
    redis.password-fileREDIS_PASSWORD_FILEPassword file of the Redis instance to scrape, defaults to "" (no password file).
    check-keysREDIS_EXPORTER_CHECK_KEYSComma separated list of key patterns to export value and length/size, eg: db3=user_count will export key user_count from db 3. db defaults to 0 if omitted. The key patterns specified with this flag will be found using SCAN. Use this option if you need glob pattern matching; check-single-keys is faster for non-pattern keys. Warning: using --check-keys to match a very large number of keys can slow down the exporter to the point where it doesn't finish scraping the redis instance.
    check-single-keysREDIS_EXPORTER_CHECK_SINGLE_KEYSComma separated list of keys to export value and length/size, eg: db3=user_count will export key user_count from db 3. db defaults to 0 if omitted. The keys specified with this flag will be looked up directly without any glob pattern matching. Use this option if you don't need glob pattern matching; it is faster than check-keys.
    check-streamsREDIS_EXPORTER_CHECK_STREAMSComma separated list of stream-patterns to export info about streams, groups and consumers. Syntax is the same as check-keys.
    check-single-streamsREDIS_EXPORTER_CHECK_SINGLE_STREAMSComma separated list of streams to export info about streams, groups and consumers. The streams specified with this flag will be looked up directly without any glob pattern matching. Use this option if you don't need glob pattern matching; it is faster than check-streams.
    check-keys-batch-sizeREDIS_EXPORTER_CHECK_KEYS_BATCH_SIZEApproximate number of keys to process in each execution. This is basically the COUNT option that will be passed into the SCAN command as part of the execution of the key or key group metrics, see COUNT option. Larger value speeds up scanning. Still Redis is a single-threaded app, huge COUNT can affect production environment.
    count-keysREDIS_EXPORTER_COUNT_KEYSComma separated list of patterns to count, eg: db3=sessions:* will count all keys with prefix sessions: from db 3. db defaults to 0 if omitted. Warning: The exporter runs SCAN to count the keys. This might not perform well on large databases.
    scriptREDIS_EXPORTER_SCRIPTPath to Redis Lua script for gathering extra metrics.
    debugREDIS_EXPORTER_DEBUGVerbose debug output
    log-formatREDIS_EXPORTER_LOG_FORMATLog format, valid options are txt (default) and json.
    namespaceREDIS_EXPORTER_NAMESPACENamespace for the metrics, defaults to redis.
    connection-timeoutREDIS_EXPORTER_CONNECTION_TIMEOUTTimeout for connection to Redis instance, defaults to "15s" (in Golang duration format)
    web.listen-addressREDIS_EXPORTER_WEB_LISTEN_ADDRESSAddress to listen on for web interface and telemetry, defaults to 0.0.0.0:9121.
    web.telemetry-pathREDIS_EXPORTER_WEB_TELEMETRY_PATHPath under which to expose metrics, defaults to /metrics.
    redis-only-metricsREDIS_EXPORTER_REDIS_ONLY_METRICSWhether to also export go runtime metrics, defaults to false.
    include-config-metricsREDIS_EXPORTER_INCL_CONFIG_METRICSWhether to include all config settings as metrics, defaults to false.
    include-system-metricsREDIS_EXPORTER_INCL_SYSTEM_METRICSWhether to include system metrics like total_system_memory_bytes, defaults to false.
    redact-config-metricsREDIS_EXPORTER_REDACT_CONFIG_METRICSWhether to redact config settings that include potentially sensitive information like passwords.
    ping-on-connectREDIS_EXPORTER_PING_ON_CONNECTWhether to ping the redis instance after connecting and record the duration as a metric, defaults to false.
    is-tile38REDIS_EXPORTER_IS_TILE38Whether to scrape Tile38 specific metrics, defaults to false.
    is-clusterREDIS_EXPORTER_IS_CLUSTERWhether this is a redis cluster (Enable this if you need to fetch key level data on a Redis Cluster).
    export-client-listREDIS_EXPORTER_EXPORT_CLIENT_LISTWhether to scrape Client List specific metrics, defaults to false.
    export-client-portREDIS_EXPORTER_EXPORT_CLIENT_PORTWhether to include the client's port when exporting the client list. Warning: including the port increases the number of metrics generated and will make your Prometheus server take up more memory
    skip-tls-verificationREDIS_EXPORTER_SKIP_TLS_VERIFICATIONWhether to to skip TLS verification
    tls-client-key-fileREDIS_EXPORTER_TLS_CLIENT_KEY_FILEName of the client key file (including full path) if the server requires TLS client authentication
    tls-client-cert-fileREDIS_EXPORTER_TLS_CLIENT_CERT_FILEName the client cert file (including full path) if the server requires TLS client authentication
    tls-server-key-fileREDIS_EXPORTER_TLS_SERVER_KEY_FILEName of the server key file (including full path) if the web interface and telemetry should use TLS
    tls-server-cert-fileREDIS_EXPORTER_TLS_SERVER_CERT_FILEName of the server certificate file (including full path) if the web interface and telemetry should use TLS
    tls-server-ca-cert-fileREDIS_EXPORTER_TLS_SERVER_CA_CERT_FILEName of the CA certificate file (including full path) if the web interface and telemetry should require TLS client authentication
    tls-server-min-versionREDIS_EXPORTER_TLS_SERVER_MIN_VERSIONMinimum TLS version that is acceptable by the web interface and telemetry when using TLS, defaults to TLS1.2 (supports TLS1.0,TLS1.1,TLS1.2,TLS1.3).
    tls-ca-cert-fileREDIS_EXPORTER_TLS_CA_CERT_FILEName of the CA certificate file (including full path) if the server requires TLS client authentication
    set-client-nameREDIS_EXPORTER_SET_CLIENT_NAMEWhether to set client name to redis_exporter, defaults to true.
    check-key-groupsREDIS_EXPORTER_CHECK_KEY_GROUPSComma separated list of LUA regexes for classifying keys into groups. The regexes are applied in specified order to individual keys, and the group name is generated by concatenating all capture groups of the first regex that matches a key. A key will be tracked under the unclassified group if none of the specified regexes matches it.
    max-distinct-key-groupsREDIS_EXPORTER_MAX_DISTINCT_KEY_GROUPSMaximum number of distinct key groups that can be tracked independently per Redis database. If exceeded, only key groups with the highest memory consumption within the limit will be tracked separately, all remaining key groups will be tracked under a single overflow key group.
    config-commandREDIS_EXPORTER_CONFIG_COMMANDWhat to use for the CONFIG command, defaults to CONFIG.

    Redis instance addresses can be tcp addresses: redis://localhost:6379, redis.example.com:6379 or e.g. unix sockets: unix:///tmp/redis.sock.\
    SSL is supported by using the rediss:// schema, for example: rediss://azure-ssl-enabled-host.redis.cache.windows.net:6380 (note that the port is required when connecting to a non-standard 6379 port, e.g. with Azure Redis instances).\

    Command line settings take precedence over any configurations provided by the environment variables.

    Authenticating with Redis

    If your Redis instance requires authentication then there are several ways how you can supply
    a username (new in Redis 6.x with ACLs) and a password.

    You can provide the username and password as part of the address, see here for the official documentation of the redis:// scheme.
    You can set -redis.password-file=sample-pwd-file.json to specify a password file, it's used whenever the exporter connects to a Redis instance,
    no matter if you're using the /scrape endpoint for multiple instances or the normal /metrics endpoint when scraping just one instance.
    It only takes effect when redis.password == "". See the contrib/sample-pwd-file.json for a working example, and make sure to always include the redis:// in your password file entries.

    An example for a URI including a password is: redis://<<username (optional)>>:<<PASSWORD>>@<<HOSTNAME>>:<<PORT>>

    Alternatively, you can provide the username and/or password using the --redis.user and --redis.password directly to the redis_exporter.

    If you want to use a dedicated Redis user for the redis_exporter (instead of the default user) then you need enable a list of commands for that user.
    You can use the following Redis command to set up the user, just replace <<<USERNAME>>> and <<<PASSWORD>>> with your desired values.

    ACL SETUSER <<<USERNAME>>> +client +ping +info +config|get +cluster|info +slowlog +latency +memory +select +get +scan +xinfo +type +pfcount +strlen +llen +scard +zcard +hlen +xlen +eval allkeys on > <<<PASSWORD>>>

    Run via Docker

    The latest release is automatically published to the Docker registry.

    You can run it like this:

    docker run -d --name redis_exporter -p 9121:9121 oliver006/redis_exporter

    Docker images are also published to the quay.io docker repo so you can pull them from there if for instance you run into rate limiting issues with Docker hub.

    docker run -d --name redis_exporter -p 9121:9121 quay.io/oliver006/redis_exporter

    The latest docker image contains only the exporter binary.
    If e.g. for debugging purposes, you need the exporter running
    in an image that has a shell then you can run the alpine image:

    docker run -d --name redis_exporter -p 9121:9121 oliver006/redis_exporter:alpine

    If you try to access a Redis instance running on the host node, you'll need to add --network host so the
    redis_exporter container can access it:

    docker run -d --name redis_exporter --network host oliver006/redis_exporter

    Run on Kubernetes

    Here is an example Kubernetes deployment configuration for how to deploy the redis_exporter as a sidecar to a Redis instance.

    Tile38

    Tile38 now has native Prometheus support for exporting server metrics and basic stats about number of objects, strings, etc.
    You can also use redis_exporter to export Tile38 metrics, especially more advanced metrics by using Lua scripts or the -check-keys flag.\
    To enable Tile38 support, run the exporter with --is-tile38=true.

    What's exported

    Most items from the INFO command are exported,
    see Redis documentation for details.\
    In addition, for every database there are metrics for total keys, expiring keys and the average TTL for keys in the database.\
    You can also export values of keys by using the -check-keys (or related) flag. The exporter will also export the size (or, depending on the data type, the length) of the key.
    This can be used to export the number of elements in (sorted) sets, hashes, lists, streams, etc.
    If a key is in string format and matches with --check-keys (or related) then its string value will be exported as a label in the key_value_as_string metric.

    If you require custom metric collection, you can provide a Redis Lua script using the -script flag. An example can be found in the contrib folder.

    The redis_memory_max_bytes metric

    The metric redis_memory_max_bytes will show the maximum number of bytes Redis can use.\
    It is zero if no memory limit is set for the Redis instance you're scraping (this is the default setting for Redis).\
    You can confirm that's the case by checking if the metric redis_config_maxmemory is zero or by connecting to the Redis instance via redis-cli and running the command CONFIG GET MAXMEMORY.

    What it looks like

    Example Grafana screenshots:
    redis_exporter_screen_01

    redis_exporter_screen_02

    Grafana dashboard is available on grafana.com and/or github.com.

    Viewing multiple Redis simultaneously

    If running Redis Sentinel, it may be desirable to view the metrics of the various cluster members simultaneously. For this reason the dashboard's drop down is of the multi-value type, allowing for the selection of multiple Redis. Please note that there is a caveat; the single stat panels up top namely uptime, total memory use and clients do not function upon viewing multiple Redis.

    Using the mixin

    There is a set of sample rules, alerts and dashboards available in redis-mixin

    Upgrading from 0.x to 1.x

    PR #256 introduced breaking changes which were released as version v1.0.0.

    If you only scrape one Redis instance and use command line flags --redis.address
    and --redis.password then you're most probably not affected.
    Otherwise, please see PR #256 and this README for more information.

    Memory Usage Aggregation by Key Groups

    When a single Redis instance is used for multiple purposes, it is useful to be able to see how Redis memory is consumed among the different usage scenarios. This is particularly important when a Redis instance with no eviction policy is running low on memory as we want to identify whether certain applications are misbehaving (e.g. not deleting keys that are no longer in use) or the Redis instance needs to be scaled up to handle the increased resource demand. Fortunately, most applications using Redis will employ some sort of naming conventions for keys tied to their specific purpose such as (hierarchical) namespace prefixes which can be exploited by the check-keys, check-single-keys, and count-keys parameters of redis_exporter to surface the memory usage metrics of specific scenarios. Memory usage aggregation by key groups takes this one step further by harnessing the flexibility of Redis LUA scripting support to classify all keys on a Redis instance into groups through a list of user-defined LUA regular expressions so memory usage metrics can be aggregated into readily identifiable groups.

    To enable memory usage aggregation by key groups, simply specify a non-empty comma-separated list of LUA regular expressions through the check-key-groups redis_exporter parameter. On each aggregation of memory metrics by key groups, redis_exporter will set up a SCAN cursor through all keys for each Redis database to be processed in batches via a LUA script. Each key batch is then processed by the same LUA script on a key-by-key basis as follows:

    1. The MEMORY USAGE command is called to gather memory usage for each key
    2. The specified LUA regexes are applied to each key in the specified order, and the group name that a given key belongs to will be derived from concatenating the capture groups of the first regex that matches the key. For example, applying the regex ^(.*)_[^_]+$ to the key key_exp_Nick would yield a group name of key_exp. If none of the specified regexes matches a key, the key will be assigned to the unclassified group

    Once a key has been classified, the memory usage and key counter for the corresponding group will be incremented in a local LUA table. This aggregated metrics table will then be returned alongside the next SCAN cursor position to redis_exporter when all keys in a batch have been processed, and redis_exporter can aggregate the data from all batches into a single table of grouped memory usage metrics for the Prometheus metrics scrapper.

    Besides making the full flexibility of LUA regex available for classifying keys into groups, the LUA script also has the benefit of reducing network traffic by executing all MEMORY USAGE commands on the Redis server and returning aggregated data to redis_exporter in a far more compact format than key-level data. The use of SCAN cursor over batches of keys processed by a server-side LUA script also helps prevent unbounded latency bubble in Redis's single processing thread, and the batch size can be tailored to specific environments via the check-keys-batch-size parameter.

    Scanning the entire key space of a Redis instance may sound a lttle extravagant, but it takes only a single scan to classify all keys into groups, and on a moderately sized system with ~780K keys and a rather complex list of 17 regexes, it takes an average of ~5s to perform a full aggregation of memory usage by key groups. Of course, the actual performance for specific systems will vary widely depending on the total number of keys, the number and complexity of regexes used for classification, and the configured batch size.

    To protect Prometheus from being overwhelmed by a large number of time series resulting from misconfigured group classification regular expression (e.g. applying the regular expression ^(.*)$ where each key will be classified into its own distinct group), a limit on the number of distinct key groups per Redis database can be configured via the max-distinct-key-groups parameter. If the max-distinct-key-groups limit is exceeded, only the key groups with the highest memory usage within the limit will be tracked separately, remaining key groups will be reported under a single overflow key group.

    Here is a list of additional metrics that will be exposed when memory usage aggregation by key groups is enabled:

    NameLabelsDescription
    redis_key_group_countdb,key_groupNumber of keys in a key group
    redis_key_group_memory_usage_bytesdb,key_groupMemory usage by key group
    redis_number_of_distinct_key_groupsdbNumber of distinct key groups in a Redis database when the overflow group is fully expanded
    redis_last_key_groups_scrape_duration_millisecondsDuration of the last memory usage aggregation by key groups in milliseconds

    Script to collect Redis lists and respective sizes.

    If using Redis version < 4.0, most of the helpful metrics which we need to gather based on length or memory is not possible via default redis_exporter.
    With the help of LUA scripts, we can gather these metrics.
    One of these scripts contrib/collect_lists_length_growing.lua will help to collect the length of redis lists.
    With this count, we can take following actions such as Create alerts or dashboards in Grafana or any similar tools with these Prometheus metrics.

    Development

    The tests require a variety of real Redis instances to not only verify correctness of the exporter but also
    compatibility with older versions of Redis and with Redis-like systems like KeyDB or Tile38.\
    The contrib/docker-compose-for-tests.yml file has service definitions for
    everything that's needed.\
    You can bring up the Redis test instances first by running make docker-env-up and then, every time you want to run the tests, you can run make docker-test. This will mount the current directory (with the .go source files) into a docker container and kick off the tests.\
    Once you're done testing you can bring down the stack by running make docker-env-down.\
    Or you can bring up the stack, run the tests, and then tear down the stack, all in one shot, by running make docker-all.

    Note. Tests initialization can lead to unexpected results when using a persistent testing environment. When make docker-env-up is executed once and make docker-test is constantly run or stopped during execution, the number of keys in the database changes, which can lead to unexpected failures of tests. Use make docker-env-down periodacally to clean up as a workaround.

    Communal effort

    Open an issue or PR if you have more suggestions, questions or ideas about what to add.

  • Redis Exporter Helm Chart
  • Redis Exporter Helm Chart

    The exporter, alert rule, and dashboard can be deployed in Kubernetes using the Helm chart. The Helm chart used for deployment is taken from the Prometheus community, which can be found here.

    Installing Redis cluster

    If your Redis cluster is not up and ready, you can start it using Helm:

    $ helm repo add bitnami https://charts.bitnami.com/bitnami
    $ helm install my-release  bitnami/redis --set master.extraFlags={"--maxmemory 1gb"}

    Note that bitnami charts allow you to start a Redis exporter as a side car for the Redis container. You can enable that by adding “--set metrics.enabled=true”

    Installing Redis Exporter
    helm repo add Prometheus-community https://prometheus-community.github.io/helm-charts
    
    helm repo update
    helm install my-release prometheus-community/prometheus-redis-exporter

    Some of the common parameters that must be changed in the values file include: 

    redisAddress: "redis://redis-master:6379"
    Auth:
      enabled: true
      redisPassword: secretpassword 
    

    All these parameters can be tuned via the values.yaml file here.

    Scrape the metrics

    There are multiple ways to scrape the metrics as discussed above. In addition to the native way of setting up Prometheus monitoring, a service monitor can be deployed (if a Prometheus operator is being used) to scrap the data from the Redis exporter. With this approach, multiple Redis servers can be scrapped without altering the Prometheus configuration. Every Redis exporter comes with its own service monitor.
    In the above-mentioned chart, a service monitor can be deployed by turning it on from the values.yaml file here.

    serviceMonitor:
      # When set true then use a ServiceMonitor to configure scraping
      enabled: true
      # Set the namespace the ServiceMonitor should be deployed
      # namespace: monitoring
      # Set how frequently Prometheus should scrape
      # interval: 30s
      # Set path to redis-exporter telemtery-path
      # telemetryPath: /metrics
      # Set labels for the ServiceMonitor, use this to define your scrape label for Prometheus Operator
      # labels:
      # Set timeout for scrape
      # timeout: 10s
      # Set relabel_configs as per https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
      # relabelings: []
      # Set of labels to transfer on the Kubernetes Service onto the target.
      # targetLabels: []
      # metricRelabelings: []

    Update the annotation section here if not using the Prometheus Operator.

    service: 
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/scrape: "true"

    This concludes our review of the Redis exporter! If you have any questions, you can reach to us via support@nexclipper.io for further discussions. Stay tuned for more useful exporter reviews and other tips coming soon.

  • Redis Exporter Alerts
  • Redis Exporter Alerts

    After digging into all the valuable metrics, this section explains in detail how we can get critical alerts.

    PromQL is a query language for the Prometheus monitoring system. It is designed for building powerful yet simple queries for graphs, alerts, or derived time series (aka recording rules). PromQL is designed from scratch and has zero common grounds with other query languages used in time series databases, such as SQL in TimescaleDB, InfluxQL, or Flux. More details can be found here.

    Prometheus comes with a built-in Alert Manager that is responsible for sending alerts (could be email, Slack, or any other supported channel) when any of the trigger conditions is met. Alerting rules allow users to define alerts based on Prometheus query expressions. They are defined based on the available metrics scraped by the exporter. Click here for a good source for community-defined alerts.

    A general alert looks as follows:

    - alert:(Alert Name)
    expr: (Metric exported from exporter) >/</==/<=/=> (Value)
    for: (wait for a certain duration between first encountering a new expression output vector element and counting an alert as firing for this element)
    labels: (allows specifying a set of additional labels to be attached to the alert)
    annotation: (specifies a set of informational labels that can be used to store longer additional information)

    Some of the recommended Redit alerts are:

    Alert - Redis is down

     - alert: RedisDown
        expr: redis_up == 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: Redis down (instance {{ $labels.instance }})
          description: "Redis instance is down\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

    Alert - Redis out of memory

    # The exporter must be started with --include-system-metrics flag or REDIS_EXPORTER_INCL_SYSTEM_METRICS=true environment variable.
      - alert: RedisOutOfSystemMemory
        expr: redis_memory_used_bytes / redis_total_system_memory_bytes * 100 > 90
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Redis out of system memory (instance {{ $labels.instance }})
          description: "Redis is running out of system memory (> 90%)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

    ➡ Alert - Too many connections

    - alert: RedisTooManyConnections
        expr: redis_connected_clients > 100
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Redis too many connections (instance {{ $labels.instance }})
          description: "Redis instance has too many connections\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

     Alert -  Redis rejecting connections

     - alert: RedisRejectedConnections
        expr: increase(redis_rejected_connections_total[1m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: Redis rejected connections (instance {{ $labels.instance }})
          description: "Some connections to Redis has been rejected\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

    ➡ Alert - Redis out of system memory

    # The exporter must be started with --include-system-metrics flag or REDIS_EXPORTER_INCL_SYSTEM_METRICS=true environment variable.
      - alert: RedisOutOfSystemMemory
        expr: redis_memory_used_bytes / redis_total_system_memory_bytes * 100 > 90
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Redis out of system memory (instance {{ $labels.instance }})
          description: "Redis is running out of system memory (> 90%)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
    

    Redis missing master

     - alert: RedisMissingMaster
        expr: (count(redis_instance_info{role="master"}) or vector(0)) < 1
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: Redis missing master (instance {{ $labels.instance }})
          description: "Redis cluster has no node marked as master.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

    Redis disconnected slaves

     - alert: RedisDisconnectedSlaves
        expr: count without (instance, job) (redis_connected_slaves) - sum without (instance, job) (redis_connected_slaves) - 1 > 1
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: Redis disconnected slaves (instance {{ $labels.instance }})
          description: "Redis not replicating for all slaves. Consider reviewing the redis replication status.\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

    Redis replication broken

     - alert: RedisReplicationBroken
        expr: delta(redis_connected_slaves[1m]) < 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: Redis replication broken (instance {{ $labels.instance }})
          description: "Redis instance lost a slave\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
  • Redis Exporter Grafana
  • Redis Exporter Grafana

    Graphs are easier to understand and more user-friendly than a row of numbers. For this purpose, users can plot their time series data in visualized format using Grafana.

    Grafana is an open-source dashboarding tool used for visualizing metrics with the help of customizable and illustrative charts and graphs. It connects very well with Prometheus and makes monitoring easy and informative. Dashboards in Grafana are made up of panels, with each panel running a PromQL query to fetch metrics from Prometheus.
    Grafana supports community-driven graphs for most of the widely used software, which can be directly imported to the Grafana Community.

    NexClipper uses the Redis by the downager dashboard, which is widely accepted and has a lot of useful panels.

    What is a Panel?

    Panels are the most basic component of a dashboard and can display information in various ways, such as gauge, text, bar chart, graph, and so on. They provide information in a very interactive way. Users can view every panel separately and check the value of metrics within a specific time range. 
    The values on the panel are queried using PromQL, which is Prometheus Query Language. PromQL is a simple query language used to query metrics within Prometheus. It enables users to query data, aggregate and apply arithmetic functions to the metrics, and then further visualize them on panels.

    Here are some examples of panels:

0 0 votes
Article Rating

Leave a Reply

1 Comment
Inline Feedbacks
View all comments
© 2022 ExporterHub.io