Skip to content

Splunk Cloud

Overview

Splunk Cloud stores and searches WxCDI syslog, traps, events, and CUCM CDR/CMR data. US/APAC and EMEA use separate documented instances.

graph LR
    Devices["WxCDI Devices"] --> Collectors["Syslog / Trap / CDR Collectors"]
    Inventory["Inventory Metadata"] --> Collectors
    Collectors --> Files["Structured Local Files"]
    Files --> UF["Splunk Universal Forwarder"]
    UF --> Cloud["Regional Splunk Cloud"]
    Cloud --> Search["Search and Dashboards"]
    Cloud --> Alerts["Scheduled Alerts"]
    Alerts --> EMS["KeepHQ / EMS"]

Use the Splunk Cloud logging page for regional instance links and the current index catalog.

Index Management

Current indexes and their intended data are listed in Logging - Splunk Cloud. A new index requires a justified difference in retention, classification, access, residency, capacity ownership, or ingestion behavior.

Retention values are not documented here because they were not verified in the reviewed repositories.

When data is missing, search by the broadest reliable source fields before assuming loss. A wrong index, source type, host normalization, regional destination, or event-time extraction can make ingested data appear absent.

index=<expected_index> earliest=-30m
| stats count latest(_time) AS latest_event BY host sourcetype
| convert ctime(latest_event)

Use the exact field names present in the target index and avoid unbounded all-time searches.

Ingestion Path

The documented syslog pipeline uses syslog-ng to receive, parse, normalize, enrich, and write device events to structured filesystem paths. Splunk Universal Forwarder monitors those paths and routes each device class to the intended index. See Syslog Infrastructure.

SNMP traps follow a similar persisted-file handoff from Telegraf to Universal Forwarder. CUCM CDR/CMR uses the separately managed SFTP metadata and ingestion path.

Alerts as Code

Production scheduled searches are managed in Terraform rather than as UI-only objects.

sequenceDiagram
    participant Dev as Engineer
    participant Git as Splunk Alert Repository
    participant TFE as Terraform Enterprise
    participant Splunk as Splunk Cloud
    participant EMS as KeepHQ / EMS

    Dev->>Git: Add or update alert YAML
    Git->>TFE: Generate speculative plan
    TFE-->>Git: Report validation and plan
    Dev->>Git: Merge approved change
    Git->>TFE: Apply managed run
    TFE->>Splunk: Create or update scheduled search
    Splunk->>EMS: Deliver matched alert action

A definition should make these behaviors explicit:

  • index, search window, and schedule;
  • trigger type, comparator, and threshold;
  • digest versus per-result behavior;
  • suppression fields and period;
  • severity, expiration, and enabled state;
  • notification action and required payload fields.

Test the search over positive and negative windows and verify that suppression does not combine unrelated customers. See Splunk Alerts.

Access and SSO

Splunk access uses the managed SSO and role-mapping workflow. Identity-provider metadata, certificates, ACS/entity identifiers, and emergency native-auth procedures are platform-admin responsibilities and should be verified in the live tenant before any change. Follow Splunk Access rather than duplicating credentials or admin steps here.

Apps & Add-ons

Use only platform-approved apps and add-ons. The current installed set must be retrieved from Splunk administration before creating dependencies.

Administration

Platform responsibilities include index/role governance, regional routing, parsing/timestamps, inventory enrichment, Universal Forwarder health, scheduled-search performance, capacity, and retention. Treat recurring unknown_syslog sources as an inventory/enrichment backlog.

Troubleshooting

Symptom Check
No events found Region, index, time range, host/source fields, and ingestion delay
Events land in unknown_syslog Inventory metadata, source address mapping, and device-type parser
Events arrive late Device timestamp, time-zone parsing, forwarder queue, and ingestion time
One source type stops syslog-ng/Telegraf file output and Universal Forwarder monitor stanza
Alert search returns results but sends nothing Alert action, payload fields, suppression, and EMS endpoint
Terraform and UI differ TFE apply result and manual UI drift