EMS (KeepHQ)

Overview

EMS/KeepHQ is the alert notification and event-routing layer used by selected WxCDI Grafana and Splunk alerts.

Alert Sources

Source	Type
Grafana	Metric, service-health, and scrape-health alert instances
Splunk	Scheduled log, trap, event, and CDR/CMR alerts

Notification Channels

The final destination is selected by the alert's contact point, severity, ownership, and routing configuration. Internal monitoring guidance documents EMS and Webex delivery for warning/high paths and adds PagerDuty for selected critical paths. Verify the current contact point before relying on a destination.

Webhook URLs, authentication, and receiver configuration are security-sensitive and must remain in managed configuration and secret systems.

Alert Routing

graph LR
    A[Grafana Alert Rule] --> C[Notification Policy]
    B[Splunk Alert] --> D[Alert Action]
    C --> E[Contact Point]
    D --> E
    E --> F[EMS / KeepHQ]
    F --> G[Severity and Ownership Routing]
    G --> H[Operational Destination]

Preserve customer, region, device, severity, owner, and runbook context through the payload. Group related instances to reduce noise without merging unrelated customer impact.

Administration

Routing changes require source control, review, deployment through the managed workflow, and a non-sensitive end-to-end test. Monitor both firing and resolved delivery. If a notification is missing, identify the last successful stage before changing the source rule.