Change-Driven Alert Suppression in EMS ¶

Design Document · EMS Platform Team

📝 Draft  |  🎯 Scope: PCC Team Alert Triage  |  🛠 EMS + EAP Change Agent

Contents ¶

1. Background
2. System Architecture Overview
3. Problem Statement
4. Solution Design
5. Scope
6. Decision Rationale
7. Flow Diagram
8. EMS Required Changes
9. Operator Workflow Change
10. Success Metrics

1. Background ¶

During planned maintenance changes (ServiceNow Change Requests), related alerts flood the EMS feed, causing noise for the PCC team. Today there is no automated mechanism to suppress these expected alerts for the duration of the change window.

2. System Architecture Overview ¶

┌─────────────────────────────────┐       ┌─────┐       ┌────────────────────┐
│         ALERT SOURCES           │       │     │       │       USERS        │
│  Grafana / Prometheus AlertMgr  │       │     │       │  PCC Team          │
│  Splunk                         │  →→→  │ EMS │  →→→  │  Service Ops Teams │
│  TAP Tests                      │       │     │       │  (Alert triage)    │
│  EM7                            │       │     │       └────────────────────┘
│  ThousandEyes                   │       └─────┘
│  SolarWinds                     │
└─────────────────────────────────┘

3. Problem Statement ¶

When a service owner executes a planned change (e.g. a network or infrastructure maintenance), a burst of expected alerts is generated by the upstream monitoring systems and lands in the EMS feed. Triage owners must manually distinguish these expected change-related alerts from genuine incidents, increasing triage time and cognitive load.

There is currently no automated suppression tied to the change lifecycle. The desired behaviour is: when a ServiceNow change enters Implement status, the alerts that are predictably caused by that change are automatically suppressed in EMS for the duration of the change window.

4. Solution Design ¶

4.1 Trigger ¶

EMS subscribes to all ServiceNow change-status events as the single source of truth for suppression triggers — regardless of whether the status was updated via EAP or directly in ServiceNow. When a change transitions to Implement, EMS directly creates a Maintenance Window in code. No external post-step call is needed.

4.2 Maintenance Window Creation ¶

When the Implement event is received, EMS first decides the window mode, then builds the CEL expression accordingly. The mode depends on whether the change type has an explicit change-type ↔ alert-name mapping (see §4.3):

• Suppress mode (mapping exists — high confidence) — the window actively suppresses matching alerts. Its CEL is more precise: customer + device plus alert_name ∈ mapped set, so only the alert names this change type is known to produce are caught.
• Label-only mode (no mapping — low confidence) — the window does not suppress; it only labels and groups. Its CEL is broader: customer + device only, since we cannot narrow by alert name.

Label-only is a new Maintenance Window mode

Rather than introducing a separate correlation/enrichment mechanism, label-only is added as an option to the existing Maintenance Window feature: it injects a label and groups matching alerts without changing their status. This keeps the whole flow terminating at a single point — create a Maintenance Window — regardless of confidence.

The window schedule (start_time / end_time) is set from the change's planned start and end dates in both modes.

CEL building method:

• Phase 1: static rule-based build directly from change metadata (e.g. customer == 'Acme Corp', labels.device_name == 'web-server-01', and in suppress mode labels.alertname in [...]). The CI list in the ticket is expected to already be comprehensive at this point (see note below).
• Phase 2 (future): an AI Agent builds the CEL from broader change context, enabling richer and more accurate scope inference in either mode. Like Multi-Tier dependency traversal for CI discovery from topology.

4.3 Alert Handling: Suppress vs. Label-only ¶

Upstream: CI coverage at ticket creation time

Topology-based CI expansion (discovering VMs running on an affected host, peer/cluster members not explicitly listed, etc.) should happen when the change ticket is being created, not at suppression time. The affected CI list in ServiceNow should already be complete and accurate by the time the ticket reaches Implement status.

A CEL match by customer + device alone is a fairly broad signal — it means "an alert on a device touched by this change", not necessarily "an alert caused by this change". Suppressing every such alert risks over-suppression: hiding a genuine, unrelated incident that happens to fire on the same device during the change window. To guard against this, EMS chooses the mode based on confidence:

Suppress mode (high confidence). Only when the change type has an explicit change-type ↔ alert-name mapping — i.e. we know which specific alert names this kind of change is expected to produce — does EMS build a suppressing window whose CEL includes that alert-name set. A matching alert is marked with status = SUPPRESSED, tagged with the change_number label, grouped under the change, and removed from the operator's default view.

Label-only mode (low confidence). When no such mapping exists, EMS builds a label-only window scoped to customer + device only. A matching alert is processed normally (stays visible for triage), but is tagged with a suspected_change_number label and grouped under the change. This gives the operator the context "a change is running on this device, this alert may be related" without risking that a real incident is silently hidden.

Label naming

change_number and suspected_change_number are deliberately distinct keys. The former asserts the alert is change-caused (and was suppressed); the latter only flags a suspected correlation (device + time overlap) with no suppression. Both carry the same CHGxxxx value, so a single query can still retrieve everything related to a change while preserving the confidence distinction.

Alerts that do not match any active window's CEL are processed normally.

4.4 Early Window Termination ¶

When an operator sets a change to Complete, Cancel, or Review in ServiceNow before its scheduled end time, EMS detects the status-change event via the same subscription and deactivates the corresponding Maintenance Window by setting end_time = now(). All subsequent alerts then resume normal processing.

4.5 Phase 2 — AI-assisted CEL Build ¶

In Phase 2, the static CEL build step is replaced by an AI Agent that derives the CEL expression from broader change context (ticket description, CMDB relationships, historical patterns, etc.). This enables more accurate and comprehensive suppression scope without requiring operators to manually maintain an exhaustive affected-CI list. Phase 2 is out of scope for the current implementation.

5. Scope ¶

In scope

This suppression mechanism serves the PCC team alert triage workflow in EMS. It suppresses alerts within the EMS feed for the duration of a planned change window.

Out of scope

• Suppression in Control Hub or any other downstream alert consumer. Each system's suppression remains that system's own responsibility.
• A unified / cross-team "enterprise suppression platform." This design deliberately solves the current PCC pain point without trying to build a big, all-encompassing solution that would take significantly longer to deliver and get buy-in for.

6. Decision Rationale ¶

Why not suppress upstream (Grafana / Splunk)? ¶

Silencing rules in Grafana AlertManager or Splunk requires per-rule action, with no centralised view of what is silenced and why. Mapping a change's affected devices to the correct Grafana/Splunk alert rules is operationally fragile and does not scale as the number of rules and changes grows. It also leaves the noise problem unsolved for any additional alert sources that onboard to EMS in the future.

Why not add a dedicated filter/relay layer between upstream and EMS? ¶

Inserting a new filtering layer (similar to an "AlertHub") would likely face objection from the LMA team, who own the upstream pipeline. It would also introduce a new service to maintain, increase end-to-end latency, and create a dependency on a separate component that is outside the EMS team's operational boundary.

Why EMS Maintenance Windows? ¶

EMS already has a Maintenance Window feature (backed by CEL evaluation per incoming alert) that is purpose-built for exactly this use case. Extending it with a label-only mode (§8.2) lets both the high-confidence suppress path and the low-confidence label-only path terminate at the same point, so all logic stays within EMS where the PCC team already operates, and avoids introducing any new service or cross-team dependency.

7. Flow Diagram ¶

flowchart TD
    %% ── Upstream: change creation via EAP (separate flow) ────────────────
    subgraph CREATION ["🔧 Change Creation (via EAP Change Agent)"]
      direction TB
      CREATE_USER(["👤 Operator creates
Change Ticket in EAP"])
      TOPO_AGENT["AI Agent queries
CI / Device Topology"]
      TOPO_VMS["Discover: VMs / workloads
running on affected host"]
      TOPO_PEERS["Discover: peer / cluster members
not explicitly listed"]
      FILL_CI["Populate Affected CI list
in ServiceNow ticket"]

      CREATE_USER --> TOPO_AGENT
      TOPO_AGENT --> TOPO_VMS
      TOPO_AGENT --> TOPO_PEERS
      TOPO_VMS   --> FILL_CI
      TOPO_PEERS --> FILL_CI
    end

    %% ── Main suppression flow ────────────────────────────────────────────
    SN_USER(["👤 Operator sets Change to
'Implement' in ServiceNow"])
    EMS_SUB["EMS subscribes to all
ServiceNow change-status events
(single source of truth)"]

    SN_USER -- "ServiceNow\nstatus event" --> EMS_SUB
    EMS_SUB --> MODE{"Change type has explicit
change-type ↔ alert-name
mapping?"}

    %% ── Suppress mode (high confidence) ──────────────────────────────────
    MODE -- "Yes\n(high confidence)" --> MW_S["Create Maintenance Window
mode = SUPPRESS"]
    MW_S --> CEL_S["Build CEL
customer + device
+ alert_name ∈ mapped set"]

    %% ── Label-only mode (low confidence) ─────────────────────────────────
    MODE -- "No\n(avoid over-suppression)" --> MW_L["Create Maintenance Window
mode = LABEL-ONLY"]
    MW_L --> CEL_L["Build CEL
customer + device"]

    subgraph PHASE2["⚡ Phase 2 — AI-assisted CEL Build (future)"]
      direction TB
      AI_CEL_BUILD["AI Agent builds CEL
from broader change context
For example: Multi-Tier
dependency traversal for
CI discovery from topology"]
    end
    CEL_S -. "Phase 2 alt" .-> AI_CEL_BUILD
    CEL_L -. "Phase 2 alt" .-> AI_CEL_BUILD

    CEL_S --> SCHED["Set Window Schedule
from Change Schedule"]
    CEL_L --> SCHED
    SCHED --> WIN

    WIN{{"Maintenance Window
Active in EMS"}}

    WIN --> MATCH{"Incoming alert
matches window CEL?"}
    MATCH -- No --> NORMAL["Alert processed normally
(stored, triaged, etc.)"]
    MATCH -- "Yes
(suppress-mode window)" --> SUPPRESSED["Alert enters EMS
with status = SUPPRESSED
+ change_number label injected
+ grouped by change
✓ Removed from default view
✓ Audit trail written"]
    MATCH -- "Yes
(label-only window)" --> LABELED["Alert processed normally
(NOT suppressed)
+ suspected_change_number label injected
+ grouped by change
✓ Stays visible for triage
✓ Audit trail written"]

    %% ── Early termination ────────────────────────────────────────────────
    CLOSE_USER(["👤 Operator sets Change to
'Complete / Cancel / Review'
in ServiceNow"])
    CLOSE_EMS["EMS subscription detects
status change event"]
    CLOSE_ACTION["Deactivate Maintenance Window
set end_time = now()"]

    CLOSE_USER -- "ServiceNow\nstatus event" --> CLOSE_EMS
    CLOSE_EMS --> CLOSE_ACTION
    CLOSE_ACTION --> MW_END(["Window terminated early
Alerts resume 
normal processing"])

8. EMS Required Changes ¶

9. Operator Workflow Change ¶

Before (today) ¶

Operators manually add a ticket_number=CHGxxxx label to alerts they believe are change-related, then manually group those alerts into a Group Alert.

After (with this solution) ¶

EMS auto-injects a change label on matching alerts and automatically groups them under the corresponding change's Group Alert. Operators no longer need to do this manually for alerts within scope.

• In suppress mode, alerts get the change_number label, are set to Suppressed, and are removed from the operator's default view — reducing noise and cognitive load during triage.
• In label-only mode, alerts get the suspected_change_number label and are grouped, but stay visible so a genuine incident is never silently hidden.

Manual fallback for missed alerts ¶

When an operator identifies an alert that was not automatically suppressed but is clearly change-related, they can:

• Manually add the change_number=CHGxxxx label to the alert.
• Batch-update the alert status to Suppressed.
• Add the alert to the corresponding change Group Alert.

Why this matters ¶

Every manually tagged alert is a signal that the auto-suppression scope missed something. Aggregating manual and automatic suppressions together lets the team:

• Identify coverage gaps and iterate on CEL rules — or generate AI skills to close them.
• Measure total suppression volume (auto + manual) and the auto hit rate.
• Quantify ops effort saved, building the business case for continued investment.

10. Success Metrics ¶

All metrics below are derivable from the change_number / suspected_change_number labels automatically injected on matching alerts (§8.5) plus the manual-fallback tagging described in §9.

Ops Effort Saved ¶

• Estimated triage hours eliminated — suppressed alert count × average time an operator would spend triaging a change-related alert. This is the headline ROI figure to present to stakeholders.
• Manual labeling toil removed — volume of alerts that would previously have required a human to add ticket_number=CHGxxxx and group manually, now handled automatically.

CEL Coverage Accuracy ¶

• Auto-suppression coverage rate — auto-suppressed alerts ÷ (auto-suppressed + manually-tagged missed alerts). Tracks how well the CEL rules capture the actual change scope. A declining rate signals rule drift; a persistently low rate for a change type justifies building a Phase 2 AI skill for that category.
• Mapping maturity / promotion candidates — suspected_change_number alerts whose alert names recur consistently for a given change type are candidates to promote into the change-type ↔ alert-name mapping, converting label-only handling into full suppression over time. The volume of suspected-but-not-suppressed alerts is the backlog that drives mapping expansion.

Team Performance During Change Windows ¶

• MTTA / MTTR for genuine incidents — mean time to acknowledge and resolve alerts that are not change-related, measured during active change windows before vs. after suppression is enabled. Fewer distractions from noise should produce a measurable drop.
• Change noise ratio — change-related alerts as a percentage of all alerts fired during a window. Quantifies the scale of the problem and the magnitude of improvement.

Auditability & Compliance ¶

• Post-change review completeness — after a change closes, every alert suppressed during the window is queryable by change number. Supports CAB reviews, SLA reporting, and incident retrospectives without manual reconstruction.

Implementation note

Add change_number and suspected_change_number as filterable labels in EMS. The existing facet / preset system can then expose a Change preset automatically once the labels are present.