Skip to content

Operational Best Practices

Dashboard Usage

  • Confirm time range, variables, and latest sample timestamp.
  • Compare a known-good peer before interpreting an absolute value.
  • Use raw queries to distinguish missing data from a dashboard-filter problem.
  • Preserve dashboard links with filters and time range in incident notes.

Alert Response

  • Distinguish service alerts from scrape/telemetry alerts.
  • Inspect alert-instance labels; one rule may represent many targets.
  • Preserve the original state before acknowledging, silencing, or editing a rule.
  • Verify notification delivery separately from alert evaluation.

Log Analysis

  • Start with the expected regional instance, index, and narrow time range.
  • Add filters one dimension at a time.
  • Use unknown_syslog to identify enrichment gaps.
  • Do not treat no search results as proof of no service event.

Incident Investigation

Correlate at least two evidence paths when customer impact is possible: metrics plus logs/traps, or service evidence plus target/scrape health. State clearly when a known coverage gap limits the conclusion.

Maintenance Windows

Identify affected targets, dashboards, rules, and ingress paths before maintenance. Use narrowly scoped, time-bounded silences and verify telemetry recovery before ending the window.