Skip to content

Telegraf

Overview

Telegraf supports pushed telemetry paths in WxCDI. The original Tools documentation identifies three deployment patterns: SNMP trap reception, Expressway collectd reception, and selected Ansible/AWX-generated metrics.

graph LR
    Traps["Devices: SNMP Traps UDP/162"] --> TrapT["Telegraf Trap Receiver"]
    TrapT --> TrapFile["Persistent Trap File"]
    TrapFile --> UF["Splunk Universal Forwarder"]
    UF --> Splunk["Splunk: snmp_traps"]

    Exp["Expressway: collectd UDP/25826"] --> Collectd["Telegraf collectd Listener"]
    AWX["AWX / Ansible Metrics"] --> Listener["Telegraf HTTPS Listener"]
    Collectd --> Mimir["Mimir"]
    Listener --> Mimir

SNMP Trap Plugin

The SNMP trap receiver runs Telegraf with Splunk Universal Forwarder in the same workload. Telegraf listens on UDP/162, parses configured MIB/OID data, enriches source information, and writes JSON events to a shared persistent path. Universal Forwarder monitors that path and sends the events to the snmp_traps index.

A representative configuration shape is:

[[inputs.snmp_trap]]
  service_address = "udp://:162"

[[outputs.file]]
  files = ["/var/log/telegraf/snmptraps.log"]
  data_format = "json"

The deployed file path, rotation, authentication, MIBs, and enrichment configuration are authoritative in the current chart and runtime values.

Expressway Collectd

The collectd listener receives Expressway vital metrics on UDP/25826. Processing normalizes the pushed data to observability conventions, including metric prefixes and labels such as product, customer, and device name.

Because collectd is pushed rather than discovered from the scrape inventory, some inventory labels may not be present in the original payload. Dashboards and alerts must handle that difference explicitly rather than assuming parity with exporter metrics.

[[inputs.socket_listener]]
  service_address = "udp://:25826"
  data_format = "collectd"

AWX and Ansible Metrics

The legacy design uses an InfluxDB v2-compatible listener for selected automation-generated metrics, then forwards the normalized metrics to Mimir. Confirm the current owning environment and endpoint before onboarding a new producer.

New producers must define authentication, payload size/rate, stable metric names and labels, ownership, retry behavior, and a way to detect dropped or rejected metrics.

Output Configuration

Trap receivers persist enriched events for the managed Splunk path. Metric receivers forward to the approved metrics backend. Exact output URLs, tenant headers, credentials, and certificates are security-sensitive and remain in managed secrets/configuration.

Deployment

Trap ingress is documented as active/active across production clusters behind resilient/Anycast ingress. Verify current load-balancer, service, image, and runtime values before making changes. Test with an approved non-sensitive trap and confirm it in Splunk.

Operational Checks

Component Monitor
Listener Packets/requests received, parse errors, rejected payloads
Processor/enrichment Unknown sources, missing labels, processor errors
File output Write errors, file size/age, rotation, filesystem capacity
Universal Forwarder Watched file, queue/backpressure, Splunk connection
Mimir output Delivery errors, retries, rejected samples, tenant routing
Kubernetes Ready replicas, restarts, service endpoints, resource saturation

Troubleshooting Sequence

  1. Confirm the expected source protocol, destination, port, and production cluster.
  2. Check the Kubernetes Service endpoints and Telegraf listener counters/logs.
  3. Confirm the payload parsed and the expected processors ran.
  4. For traps, verify the shared file and Universal Forwarder queue/output.
  5. For metrics, verify output delivery and query the latest sample in Mimir/Grafana.
  6. Compare labels with inventory and investigate unresolved sources.

See SNMP Traps, Syslog Infrastructure, and AWX.