How to set up alerting based on metrics ¶

The metrics in a Kubed cluster will be scrapped by Prometheus. And the metrics will be stored in cluster storage for 2 hours. At the same time, metrics will be sent to LMA Grafana Mimir to have a long-term storage up to 1 month. So we can set up alerting based on in-cluster metrics and LMA Grafana alerting is based on metrics stored in Mimir.

• PrometheusRule listened by Alertmanager is the alerting solution querying in-cluster metrics
• Grafana alert rule is another solution querying central metrics

PrometheusRule for in-cluster alerting ¶

In-cluster alerting is preferred to be done through Alertmanager. A default PagerDuty receiver is configured from the routing_key provided in Vault. Clusters are deployed with a default set of rules from kubed-base-alert-rules. Additional rules can be added along with application deploys by creating a PrometheusRule CRD object in any namespace. As an example, individual Kafka monitoring rules are added with the below resource.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
spec:
  groups:
  - name: kafka.rules
    rules:
    - alert: KafkaConsumerGroupLagHigh
      annotations:
        message: 'Consumer group `{{`{{ $labels.group }}`}}` for topic  `{{`{{ $labels.topic }}`}}`
                  has lag `{{`{{ printf "%0.0f" $value }}`}}` for more than {{.Values.rules.kafka.interval}}'
      expr: 'avg by (group,topic) (kafka_minion_group_topic_lag{group!="", topic!=""}) >= {{.Values.rules.kafka.limit}}'
      for: {{.Values.rules.kafka.interval}}
      labels:
        severity: webhook

LMA Grafana Alerting for central alerting ¶

Onboard Grafana alert rule via LMA Alert Gitops flow ¶

• Integration cluster need to use repo: metrics-grafana-alerts-int-commercial
• Production cluster need to use repo: metrics-grafana-alerts-prd-commercial

Detail setup flow please refer to operational doc

Onboard Grafana alert rule via Dash-tools Gitops flow ¶

Dash-tool is a primary tool to maintain and propagate LMA Grafana dashboards and alerts across multiple data-centers.