Skip to content

KeepHQ (EMS)

Overview

EMS/KeepHQ receives selected Grafana and Splunk notifications and routes them to the configured operational destinations.

graph LR
    Grafana["Grafana Alerts"] --> EMS["KeepHQ / EMS"]
    Splunk["Splunk Alerts"] --> EMS
    EMS --> Policy["Deduplication and Routing"]
    Policy --> Webex["Webex"]
    Policy --> PagerDuty["PagerDuty where approved"]
    Policy --> Other["Configured Operational Destinations"]

Integrations

Grafana contact points and Splunk alert actions provide the source events. Payloads must preserve the customer/device/region, severity, owner, and runbook context needed for investigation without exposing credentials.

At minimum, a source integration should provide a stable event identity, firing/resolved state, start time, severity, service, customer/device context, summary, evidence link, and runbook link. Test payload changes against a non-customer route before production rollout.

Notification Channels

Destinations depend on the contact point and routing policy. Internal monitoring guidance documents EMS and Webex for warning/high paths and adds PagerDuty for selected critical paths. Verify the current managed configuration before relying on a route.

Routing Rules

Routing should use stable severity and ownership fields, group related instances, deduplicate repeats, preserve resolved notifications, and avoid combining unrelated customer impact.

The legacy internal routing model sends warning/high internal alerts to EMS and Webex, and selected critical alerts to EMS, Webex, and PagerDuty. Treat the managed contact-point and routing configuration as authoritative because destination policy can change.

Delivery Validation

  1. Trigger an approved synthetic or test event.
  2. Confirm the source reports a successful webhook/action request.
  3. Verify EMS parsed the expected labels and selected the intended route.
  4. Confirm exactly one firing notification reaches each destination.
  5. Resolve the test and confirm the resolved notification.
  6. Repeat only when testing deduplication or repeat timing.

Troubleshooting

Symptom Check
No event in EMS Source contact point/action, URL, authentication, and response code
EMS receives event but sends nothing Required fields, policy match, mute window, and destination state
Wrong destination Severity/owner/contact-point labels and route precedence
Duplicate notifications Source grouping, event identity, deduplication key, and retry behavior
Resolved message missing Source send-resolved setting and EMS state correlation
Sensitive data exposed Disable unsafe payload, restrict access, rotate credentials if present, and follow incident handling

Administration

Webhook URLs, tokens, receiver configuration, and user administration are security-sensitive. Manage them through approved configuration/secret systems. Test changes end to end with non-customer data and verify both firing and resolved delivery.

See Alerting, Grafana, and Splunk.