Syslog Infrastructure

Purpose

The syslog infrastructure receives device events, enriches them with inventory context, persists them long enough for reliable handoff, and forwards them to the correct Splunk Cloud index.

Data Flow

graph LR
    subgraph Sources["WxCDI Sources"]
        VOS["CUCM / CUC / IMP / CER / Expressway"]
        Infra["ASA / CSR / Network and Infrastructure"]
    end
    VOS -->|"UDP/514"| LB["Managed Load Balancer"]
    Infra -->|"UDP/514"| LB
    LB --> Service["Kubernetes Service"]
    Service --> Syslog["syslog-ng StatefulSet"]
    Inventory["Consolidated Inventory Metadata"] --> Syslog
    Syslog --> Files["Structured Local Files"]
    Files --> UF["Splunk Universal Forwarder"]
    UF --> Splunk["Splunk Cloud Indexes"]

The production workload is documented as a StatefulSet with syslog-ng, a cleanup component, and Splunk Universal Forwarder containers. A load balancer distributes UDP/514 traffic across the service replicas.

Processing Stages

1. Receive — syslog-ng listens on the managed network source.
2. Parse — messages are parsed using the expected syslog format and device-specific rules.
3. Enrich — inventory metadata adds customer and device context.
4. Normalize — host and device type fields are rewritten to stable names.
5. Route — known device types are written to their monitored paths; unresolved sources use the unknown_syslog path/index.
6. Forward — Splunk Universal Forwarder monitors the files and sends events to Splunk Cloud.
7. Clean up — the cleanup process enforces local file lifecycle so receiver storage remains healthy.

Index Routing

Splunk monitor stanzas bind filesystem paths to an index and sourcetype. For example, firewall device logs are routed to the firewall syslog index. The authoritative mapping is the current chart/configuration; see Splunk Cloud for the documented index catalog.

Recurring unknown_syslog events mean the pipeline is receiving data but cannot map the source to expected inventory metadata. Treat this as an enrichment/inventory defect rather than hiding it with a broad parser rule.

Operational Checks

Layer	Evidence
Device	Configured destination and local send/error counters
Network	Load-balancer listener, firewall path, and traffic counters
Kubernetes	Ready replicas, endpoints, restarts, events, and storage
syslog-ng	Receive rate, parse/rewrite errors, file output, unknown sources
Cleanup	Filesystem utilization and file age
Universal Forwarder	Watched paths, queue/backpressure, connection and forwarding errors
Splunk Cloud	Latest event time, host/device fields, index/sourcetype, regional destination

Troubleshooting Sequence

1. Search Splunk broadly enough to rule out wrong index, host, or time range.
2. Compare _time and ingestion time to distinguish delay from loss.
3. Check unknown_syslog and metadata mapping for the source.
4. Verify Universal Forwarder monitoring and output health.
5. Confirm syslog-ng received and wrote the event.
6. Check service endpoints, pods, load balancer, and firewall counters.
7. Validate the device's configured destination and event generation.

See Logging architecture and Splunk for downstream search and alert operations.