Skip to content

Mimir

Overview

Mimir is the Prometheus-compatible backend receiving remote-write data from WxCDI Prometheus services and serving metric queries to Grafana.

Architecture

graph LR
    P1["Prometheus DFW"] -->|"remote_write"| Ingest["Mimir Ingestion"]
    P2["Prometheus SJC"] -->|"remote_write"| Ingest
    Ingest --> Store["Metrics Storage"]
    Store --> Query["Mimir Query Path"]
    Query --> Grafana["Grafana"]
    Query --> Rules["Alert Evaluation"]

Exact component topology and versions are owned by the shared metrics platform and current deployment configuration.

Configuration

Key integration settings include tenant/routing information, remote-write endpoint, queue behavior, authentication, and Grafana data-source configuration. Keep credentials in approved secret storage.

Prometheus and Grafana must use the same intended tenant. A healthy scrape with an empty Grafana query often indicates remote-write, tenant, data-source, or query-time-range mismatch rather than an exporter failure.

Maintenance

Monitor ingestion availability/rejections, remote-write backlog, query availability/latency, cardinality, storage, and compaction. Retention and storage policy values were not verified in the reviewed repositories; confirm them with the Mimir service owner.

Failure Isolation

Observation Likely boundary
Exporter and Prometheus are current; Mimir is stale Prometheus remote-write queue, authentication, endpoint, or Mimir ingestion
Mimir query works; Grafana panel does not Grafana data source, tenant header, variables, or panel query
Only one metric is rejected Invalid sample, timestamp, label, or series limit
Queries degrade after a rollout Cardinality growth, query shape, or backend capacity
One region is missing Regional Prometheus remote write or region-specific configuration

Operational Evidence

During an incident, capture:

  • Prometheus scrape freshness and remote-write queue/failure metrics;
  • Mimir ingestion errors or rejected-sample reason;
  • tenant and data-source identity without secret headers;
  • a bounded query showing the latest stored timestamp;
  • affected region, jobs, and approximate series count;
  • recovery time and whether backlogged samples arrived.

See Prometheus, Grafana, and Monitoring.