Skip to content

Splunk Alerts

Overview

Splunk alerts evaluate scheduled searches over syslog, traps, events, or CDR/CMR data. The managed Terraform repository is the authoritative source for production alert definitions.

Alert Rules

Each definition should include an explicit index and time window, stable routing fields, schedule, trigger condition, severity, suppression fields and period, per-result versus digest behavior, notification action, and enabled/disabled state.

Before deployment:

  1. Test the search over positive and negative windows.
  2. Confirm every selected sourcetype provides the fields used in the payload.
  3. Verify suppression prevents duplicates without combining unrelated customers.
  4. Review the Terraform plan and deploy through pull request.
  5. Test notification delivery without using real customer-sensitive data.

Notification Channel

Selected Splunk alerts send notifications through EMS/KeepHQ. Routing depends on each alert action and payload; see EMS / KeepHQ.

Creating New Alerts

Create or change alerts in the managed terraform-splunk-wxcdi-alerts repository. Avoid UI-only production changes because they create drift from Terraform.