Skip to content

Grafana Alert Rules

Overview

Grafana alert rules evaluate Mimir/Prometheus queries. One rule can create multiple alert instances—one for every unique label set returned by the query.

Alert Rules

Rules should include:

  • a service outcome or visibility failure that a responder can act on;
  • a bounded PromQL query using stable labels;
  • evaluation interval and pending duration;
  • severity, ownership, and routing labels;
  • explicit no-data and evaluation-error behavior;
  • dashboard and runbook links;
  • a contact point and tested resolved-notification behavior.

Device alert sources are referenced from the corresponding YAML files in observability-audit. The managed internal alert repository is the authoritative source for current rule names, thresholds, enabled state, and routing.

Notification Channel

Grafana rules use contact-point labels to select EMS/KeepHQ or other approved receivers. Do not assume every rule has the same final destination. See EMS / KeepHQ.

Creating New Alerts

See How-To: Create Alert Rule.

Silencing & Maintenance

Use the approved silence workflow. Match the smallest safe set of labels, include the maintenance owner/reason, and set an expiry. Verify telemetry and alert recovery before the silence ends. Never disable a shared rule as a maintenance shortcut.