Skip to content

Grafana

Overview

Grafana provides WxCDI metric dashboards, Explore, and metric alert evaluation using Mimir data.

graph LR
    Exporters["Exporters"] --> Prometheus["Prometheus"]
    Prometheus --> Mimir["Mimir"]
    Mimir --> Explore["Grafana Explore"]
    Mimir --> Dashboards["Grafana Dashboards"]
    Mimir --> Rules["Grafana Alert Rules"]
    Rules --> Policies["Notification Policies"]
    Policies --> EMS["KeepHQ / EMS"]

The shared Grafana portal is metrics.o.webex.com. Use the WxCDI folder and the approved Mimir data source; verify the current data-source UID in Grafana or the managed repository before provisioning dashboards or alerts.

Dashboard Management

Dashboard JSON should be version controlled and deployed through the managed dashboard workflow. Use stable inventory labels, bounded queries, meaningful no-data behavior, ownership/runbook links, and the established WxCDI folder/naming conventions.

A production dashboard should provide:

  • service or device scope and customer/region filters;
  • current health plus a useful historical comparison;
  • units, legends, and thresholds that an operator can interpret;
  • links from summary panels to device-level evidence;
  • an owner and source repository;
  • queries that avoid unbounded label values and expensive wide time ranges.

UI edits are useful for prototyping, but the reviewed JSON/source must remain authoritative. Export or reproduce an accepted UI change in the managed repository.

Data Sources

Mimir is the primary WxCDI metrics data source. When a panel is missing current data but the Prometheus scrape succeeds, check remote-write health, tenant routing, and the selected Grafana data source.

Use Explore to isolate dashboard problems:

  1. Select the same Mimir data source as the panel.
  2. Run the panel query over a short time range.
  3. Remove transformations and variables until the raw series is visible.
  4. Inspect labels and latest sample time.
  5. Compare the result with Prometheus only when remote-write delay is suspected.

Alerting Concepts

Concept Meaning
Alert rule Query and condition evaluated on a schedule
Alert instance One alert state for each unique label set returned by the rule
Evaluation interval How often the rule is evaluated
Notification policy Groups and routes alert instances to a contact point
Group wait Delay before the first notification for a new alert group
Group interval Delay before sending updates for an existing group
Repeat interval Time before a still-firing group is notified again

Grouping related instances reduces notification volume. Disabling grouping can create one message per series and produce an alert storm during a shared failure. Group on labels that preserve customer and service boundaries.

Alerts as Code

sequenceDiagram
    participant Dev as Engineer
    participant Repo as Managed Alert Repository
    participant CI as JSON Validation
    participant Grafana as Grafana API
    participant EMS as KeepHQ / EMS

    Dev->>Repo: Add alert rule or contact point JSON
    Repo->>CI: Validate pull request
    CI-->>Repo: Report result
    Dev->>Repo: Merge approved change
    Repo->>Grafana: Provision rule and contact point
    Grafana->>EMS: Send test firing and resolved events

Rules should include stable severity, service, owner, and routing labels; summary/description annotations; dashboard and runbook links; explicit no-data/error behavior; and tested firing/resolved delivery. See Grafana Alerts.

Plugins

The reviewed repositories do not provide a verified plugin inventory. Check current Grafana administration before adding a plugin dependency to a dashboard.

Administration

SSO, roles, folders, plugins, data sources, and global notification policies are platform-owned. Request least-privileged access and avoid UI-only changes that bypass the managed source repository.

Troubleshooting

Symptom Check
No data in one panel Panel time range, variables, data source, and raw query in Explore
Explore has no recent samples Mimir tenant/data source, remote write, and Prometheus target
Duplicate series Label joins, inventory duplication, and exporter segmentation
Alert fires once per device unexpectedly Returned label sets and notification grouping
Alert fires but no notification arrives Contact-point label, policy match, mute timing, and EMS delivery
Dashboard changed after deployment Managed JSON versus UI drift and provisioning status