Skip to content

Prometheus

Overview

Prometheus instances discover inventory targets, scrape SNMP and custom exporters, evaluate recording/rule data where configured, and remote-write samples to Mimir.

graph LR
    subgraph Devices["WxCDI Devices"]
        API["API"]
        PerfMon["PerfMon"]
        CLI["CLI / SSH"]
        SNMP["SNMP"]
    end
    API --> Exporter["Custom Exporter"]
    PerfMon --> Exporter
    CLI --> Exporter
    SNMP --> SNMPExp["SNMP Exporter"]
    Inventory["Generated Inventory Targets"] --> Prom["Prometheus"]
    Exporter -->|"scrape"| Prom
    SNMPExp -->|"scrape"| Prom
    Prom -->|"remote_write"| Mimir["Mimir"]
    Mimir --> Grafana["Grafana"]

Custom Exporter Collection

The VOS custom exporters expose device data through HTTP metric endpoints. Their configuration determines which collectors are enabled, where they apply, and how frequently they run.

Collection type Device interface Typical purpose
PerfMon SOAP PerfMon service on TCP/8443 Performance counters and gauges
API Product-specific REST or SOAP API Service, registration, version, and inventory state
CLI A shared SSH session per host NTP, backup, license, core-file, and platform state

Collection intervals are owned by each exporter's current metrics_config.yaml; do not rely on copied values in documentation. A representative definition is:

perfmon_metrics:
  <metric_name>:
    description: <operator-facing description>
    prometheus_type: Gauge
    collection_function: <function_name>
    applies_to:
      node_type: all
    enabled: true
    interval_seconds: <interval>

The metric name and labels are an API contract for dashboards and alerts. Review cardinality and downstream queries before renaming them.

PerfMon Session Flow

For CUCM-family products, the exporter uses the PerfMon SOAP workflow:

sequenceDiagram
    participant E as Exporter
    participant P as Device PerfMon Service
    E->>P: perfmonOpenSession
    P-->>E: Session handle
    E->>P: perfmonAddCounter
    E->>P: perfmonCollectSessionData
    P-->>E: Counter values
    E->>P: perfmonCloseSession

Counters are defined in exporter configuration and may be common or product/role specific. Always close sessions and avoid testing a broad counter set against production devices.

Device Exporters

The legacy tool documentation identifies custom exporters for CUCM, CUC, CER, IM&P, and Expressway. Infrastructure devices also use SNMP Exporter or product-specific collectors as documented on each device page.

Configuration

Most device targets are generated in wxcdi-inventory-files and selected through file-based service discovery and regional file globs. Scrape jobs, modules, intervals, timeouts, relabeling, remote write, and active/standby behavior are managed in the chart/config repositories.

Some exporters also require targets in wxcdi-kubed-configs; the inventory runbook specifically calls out UCS and vCenter.

Scrape configuration owns the job name, module, interval, timeout, relabeling, target file pattern, and remote-write behavior. Keep those settings aligned with the collector interval: scraping more frequently than a cached collector runs does not create fresher device data.

Targets

Device pages identify the expected Prometheus instance and exporter. A target file in Git does not prove Prometheus has consumed it. Verify git-sync/config revision, target discovery, scrape error, and latest successful sample.

Use the Targets view or Prometheus API to distinguish:

  • target was never discovered;
  • target was discovered but the scrape failed;
  • scrape succeeded but the exporter returned stale/cached data;
  • Prometheus has current data but remote write or Grafana is stale.

Useful PromQL Queries

up{job=~".*<device_or_service>.*"}

min_over_time(up{job=~".*<device_or_service>.*"}[15m]) == 0

Platform health should include discovery count, scrape failures/duration, rule evaluation, remote-write queue/failures, storage, and replica/scale state.

Troubleshooting Sequence

  1. Confirm the device is in the generated inventory and correct regional target file.
  2. Confirm Prometheus loaded the expected config and discovered the target.
  3. Inspect target health, last scrape, duration, and error.
  4. Query the exporter endpoint directly from an approved network location.
  5. Use exporter logs to identify the failing host, collector, and collection method.
  6. Test only the failing API, PerfMon request, CLI command, or SNMP module with approved credentials.
  7. Compare the raw device response with the exporter's parsed metric and labels.
  8. Confirm Prometheus ingested the new sample and remote write delivered it to Mimir.

Never paste device credentials or raw customer-sensitive responses into issues. Redact tokens, addresses, hostnames, and payload fields according to the data-handling policy.