Skip to content

Alert Manager

User can config prometheus rule and alert manager rule to trigger the notification.

Prometheus Alerting Rule

Prometheus Alerting rules allow you to define conditions based on your metrics data that, when met, will trigger alerts. These alerts are then sent to Alertmanager, which handles the alert lifecycle and notifications.

Example

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: mas-prometheus-rule
  namespace: {{ namespace }}
  labels:
    release: mas-prometheus-rule
spec:
  groups:
    - name: mct
      rules:
        - alert: MCTAlertTest
          annotations:
            message: >-
              This is a test for mct prometheus rule configuration, please ignore.
            summary: TEST, MCT cluster id 1110126 is Alerting.
          expr: mct_status_metric{mct_cluster="1110126"} == 1
          labels:
            severity: mctSeverity
            service: mct-alert
  • alert: The name of the alert (e.g., HighCPUUsage).
  • expr: The PromQL expression that defines the alert condition. The alert is triggered when this condition is true.
  • for: The duration that the condition must be true before the alert is triggered.
  • labels: Labels to attach to the alert, which can be used for routing and classification.
  • annotations: Additional information about the alert, such as a summary and description.

Alert Manger Rule

User config alert manager rule to define alert conditions and handling logic. These rules determine when an alert should be triggered based on specific conditions in the monitored data. When these conditions are met, the alerts are sent to Alertmanager for further processing and notifications. Supports various notification methods.

  • Email
  • PagerDuty
  • Webhook
  • WeChat
  • ...

Example

Alert manager rule 

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: mas-alert-manager-rule
  namespace: {{ namespace }}
  labels:
    alertmanagerConfig: mct
spec:
  route:
    groupWait: 10s
    groupInterval: 5m
    repeatInterval: 30m
    receiver: mct
    matchers:
      - name: severity
        value: mctSeverity
      - name: service
        value: mct-alert
      - name: alertname
            value: MCTAlertTest
  receivers:
    - name: mct
      pagerdutyConfigs:
        - serviceKey:
            key: {{ mas-alerting-name-key-secret }}
            name: {{ mas-alerting-name-secret }}

  • route: The default routing configuration for alerts.
  • receiver: The default receiver for alerts.
  • group_by: The labels by which alerts are grouped.
  • group_wait, group_interval, repeat_interval: Timing configurations for alert notifications.
  • routes: Specifies additional routing rules based on alert labels.
  • match: Specifies conditions to match alerts.
  • receiver: The receiver to handle the matched alerts.
  • receivers: The list of receivers.
  • name: The name of the receiver (webhook in this case).

pagerdutyConfigs 

apiVersion: v1
kind: Secret
type: Opaque
metadata:
  name: {{ mas-alerting-name-secret }}
  namespace: {{ namespace }}
data:
  {{ mas-alerting-name-key-secret }}: {{ pagerduty.integration_key }}

Conclusion

In summary, Prometheus rules are focused on defining the conditions for generating alerts based on metrics data, while Alertmanager rules are concerned with how to handle and route those alerts once they are generated.