Change-Driven Alert Suppression in EMS ¶

Design Document · EMS Platform Team

📝 Draft  |  🎯 Scope: PCC Team Alert Triage  |  🛠 EMS + EAP Change Agent

Contents ¶

1. Background
2. System Architecture Overview
3. Problem Statement
4. Solution Design
5. Scope
6. Decision Rationale
7. Flow Diagram
8. EMS Required Changes
9. Operator Workflow Change
10. Success Metrics

1. Background ¶

During planned maintenance changes (ServiceNow Change Requests), related alerts flood the EMS feed, causing noise for the PCC team. Today there is no automated mechanism to suppress these expected alerts for the duration of the change window.

2. System Architecture Overview ¶

┌─────────────────────────────────┐       ┌─────┐       ┌────────────────────┐
│         ALERT SOURCES           │       │     │       │       USERS        │
│  Grafana / Prometheus AlertMgr  │       │     │       │  PCC Team          │
│  Splunk                         │  →→→  │ EMS │  →→→  │  Service Ops Teams │
│  TAP Tests                      │       │     │       │  (Alert triage)    │
│  EM7                            │       │     │       └────────────────────┘
│  ThousandEyes                   │       └─────┘
│  SolarWinds                     │
└─────────────────────────────────┘

3. Problem Statement ¶

When a service owner executes a planned change (e.g. a network or infrastructure maintenance), a burst of expected alerts is generated by the upstream monitoring systems and lands in the EMS feed. Triage owners must manually distinguish these expected change-related alerts from genuine incidents, increasing triage time and cognitive load.

There is currently no automated suppression tied to the change lifecycle. The desired behaviour is: when a ServiceNow change enters Scheduled status, the alerts that are predictably caused by that change are automatically suppressed in EMS for the duration of the change window.

4. Solution Design ¶

4.1 Trigger ¶

EMS subscribes to all ServiceNow change-status events as the single source of truth for suppression triggers — regardless of whether the status was updated via EAP or directly in ServiceNow. When a change transitions to Scheduled, EMS directly creates a Maintenance Window in code. No external post-step call is needed.

4.2 Maintenance Window Creation ¶

When the Scheduled event is received, EMS creates a Maintenance Window with two components:

• CEL expression — built from the change's affected CI / customer fields.
  • Phase 1: static rule-based build directly from change metadata (e.g. customer == 'Acme Corp', labels.device_name == 'web-server-01'). The CI list in the ticket is expected to already be comprehensive at this point (see note below).
  • Phase 2 (future): an AI Agent builds the CEL expression from broader change context, enabling richer and more accurate scope inference.
• Window schedule — start_time and end_time are set from the change's planned start and end dates.

4.3 Alert Suppression Behaviour ¶

Upstream: CI coverage at ticket creation time

Topology-based CI expansion (discovering VMs running on an affected host, peer/cluster members not explicitly listed, etc.) should happen when the change ticket is being created, not at suppression time. The affected CI list in ServiceNow should already be complete and accurate by the time the ticket reaches Scheduled status.

While the Maintenance Window is active, any incoming alert whose fields match the CEL expression is marked with status = SUPPRESSED in EMS. Suppressed alerts remain visible in the feed but filterable, and an audit trail entry is written (change ticket label in alert body). Same change ticket alerts can be grouped together for easy triage. Alerts that do not match are processed normally.

4.4 Early Window Termination ¶

When an operator sets a change to Complete, Cancel, or Review in ServiceNow before its scheduled end time, EMS detects the status-change event via the same subscription and deactivates the corresponding Maintenance Window by setting end_time = now(). All subsequent alerts then resume normal processing.

4.5 Phase 2 — AI-assisted CEL Build ¶

In Phase 2, the static CEL build step is replaced by an AI Agent that derives the CEL expression from broader change context (ticket description, CMDB relationships, historical patterns, etc.). This enables more accurate and comprehensive suppression scope without requiring operators to manually maintain an exhaustive affected-CI list. Phase 2 is out of scope for the current implementation.

5. Scope ¶

In scope

This suppression mechanism serves the PCC team alert triage workflow in EMS. It suppresses alerts within the EMS feed for the duration of a planned change window.

Out of scope

• Suppression in Control Hub or any other downstream alert consumer. Each system's suppression remains that system's own responsibility.
• A unified / cross-team "enterprise suppression platform." This design deliberately solves the current PCC pain point without trying to build a big, all-encompassing solution that would take significantly longer to deliver and get buy-in for.

6. Decision Rationale ¶

Why not suppress upstream (Grafana / Splunk)? ¶

Silencing rules in Grafana AlertManager or Splunk requires per-rule action, with no centralised view of what is silenced and why. Mapping a change's affected devices to the correct Grafana/Splunk alert rules is operationally fragile and does not scale as the number of rules and changes grows. It also leaves the noise problem unsolved for any additional alert sources that onboard to EMS in the future.

Why not add a dedicated filter/relay layer between upstream and EMS? ¶

Inserting a new filtering layer (similar to an "AlertHub") would likely face objection from the LMA team, who own the upstream pipeline. It would also introduce a new service to maintain, increase end-to-end latency, and create a dependency on a separate component that is outside the EMS team's operational boundary.

Why EMS Maintenance Windows? ¶

EMS already has a Maintenance Window feature (backed by CEL evaluation per incoming alert) that is purpose-built for exactly this use case. Hooking the Change Agent into it requires only the changes listed in §8, keeps all suppression logic within EMS where the PCC team already operates, and avoids introducing any new service or cross-team dependency.

7. Flow Diagram ¶

flowchart TD
    %% ── Upstream: change creation via EAP (separate flow) ────────────────
    subgraph CREATION ["🔧 Change Creation (via EAP Change Agent)"]
      direction TB
      CREATE_USER(["👤 Operator creates
Change Ticket in EAP"])
      TOPO_AGENT["AI Agent queries
CI / Device Topology"]
      TOPO_VMS["Discover: VMs / workloads
running on affected host"]
      TOPO_PEERS["Discover: peer / cluster members
not explicitly listed"]
      FILL_CI["Populate Affected CI list
in ServiceNow ticket"]

      CREATE_USER --> TOPO_AGENT
      TOPO_AGENT --> TOPO_VMS
      TOPO_AGENT --> TOPO_PEERS
      TOPO_VMS   --> FILL_CI
      TOPO_PEERS --> FILL_CI
    end

    %% ── Main suppression flow ────────────────────────────────────────────
    SN_USER(["👤 Operator sets Change to
'Scheduled' in ServiceNow"])
    EMS_SUB["EMS subscribes to all
ServiceNow change-status events
(single source of truth)"]

    SN_USER -- "ServiceNow\nstatus event" --> EMS_SUB
    EMS_SUB --> MW_CREATE["Create Maintenance Window
in EMS"]

    subgraph PHASE2["⚡ Phase 2 — AI-assisted CEL Build"]
      direction TB
      AI_CEL_BUILD["AI Agent
builds CEL expression
from change context"]
    end
    MW_CREATE -- "Phase 2\n(AI-assisted)" --> AI_CEL_BUILD

    subgraph PHASE1["Phase 1 — Static CEL Build"]
      direction TB
      CEL_BUILD["Build CEL Expression
from Change Metadata"]
      CEL_CUST["Customer filter
e.g. customer == 'Acme Corp'"]
      CEL_CI["Affected CI filter
e.g. labels.device_name == 'web-server-01'"]
      CEL_BUILD --> CEL_CUST
      CEL_BUILD --> CEL_CI
    end
    MW_CREATE -- "Phase 1\n(static rules)" --> CEL_BUILD

    MW_CREATE --> SCHED["Set Window Schedule
from Change Schedule"]
    SCHED --> SCHED_OUT["start_time: change.scheduled_start
end_time: change.scheduled_end"]

    AI_CEL_BUILD --> MW_ACTIVE
    CEL_CUST     --> MW_ACTIVE
    CEL_CI       --> MW_ACTIVE
    SCHED_OUT    --> MW_ACTIVE

    MW_ACTIVE{{"Maintenance Window
Active in EMS"}}

    MW_ACTIVE --> MATCH{"Incoming alert
matches window CEL?"}
    MATCH -- No --> NORMAL["Alert processed normally
(stored, triaged, etc.)"]
    MATCH -- Yes --> SUPPRESSED["Alert enters EMS
with status = SUPPRESSED
+ change_number label injected
✓ Visible in feed
✓ Audit trail written"]

    %% ── Early termination ────────────────────────────────────────────────
    CLOSE_USER(["👤 Operator sets Change to
'Complete / Cancel / Review'
in ServiceNow"])
    CLOSE_EMS["EMS subscription detects
status change event"]
    CLOSE_ACTION["Deactivate Maintenance Window
set end_time = now()"]

    CLOSE_USER -- "ServiceNow\nstatus event" --> CLOSE_EMS
    CLOSE_EMS --> CLOSE_ACTION
    CLOSE_ACTION --> MW_END(["Window terminated early
Alerts resume normal processing"])

8. EMS Required Changes ¶

9. Operator Workflow Change ¶

Before (today) ¶

Operators manually add a ticket_number=CHGxxxx label to alerts they believe are change-related, then manually group those alerts into a Group Alert.

After (with this solution) ¶

EMS auto-injects the change_number label on suppressed alerts and automatically groups them under the corresponding change's Group Alert. Operators no longer need to do this manually for alerts within scope.

Suppressed alerts will not appear in operators' default view, reducing noise and cognitive load during triage.

Manual fallback for missed alerts ¶

When an operator identifies an alert that was not automatically suppressed but is clearly change-related, they can:

• Manually add the change_number=CHGxxxx label to the alert.
• Batch-update the alert status to Suppressed.
• Add the alert to the corresponding change Group Alert.

Why this matters ¶

Every manually tagged alert is a signal that the auto-suppression scope missed something. Aggregating manual and automatic suppressions together lets the team:

• Identify coverage gaps and iterate on CEL rules — or generate AI skills to close them.
• Measure total suppression volume (auto + manual) and the auto hit rate.
• Quantify ops effort saved, building the business case for continued investment.

10. Success Metrics ¶

All metrics below are derivable from the change_number label automatically injected on suppressed alerts (§8.3) plus the manual-fallback tagging described in §9.

Ops Effort Saved ¶

• Estimated triage hours eliminated — suppressed alert count × average time an operator would spend triaging a change-related alert. This is the headline ROI figure to present to stakeholders.
• Manual labeling toil removed — volume of alerts that would previously have required a human to add ticket_number=CHGxxxx and group manually, now handled automatically.

CEL Coverage Accuracy ¶

• Auto-suppression coverage rate — auto-suppressed alerts ÷ (auto-suppressed + manually-tagged missed alerts). Tracks how well the CEL rules capture the actual change scope. A declining rate signals rule drift; a persistently low rate for a change type justifies building a Phase 2 AI skill for that category.

Team Performance During Change Windows ¶

• MTTA / MTTR for genuine incidents — mean time to acknowledge and resolve alerts that are not change-related, measured during active change windows before vs. after suppression is enabled. Fewer distractions from noise should produce a measurable drop.
• Change noise ratio — change-related alerts as a percentage of all alerts fired during a window. Quantifies the scale of the problem and the magnitude of improvement.

Auditability & Compliance ¶

• Post-change review completeness — after a change closes, every alert suppressed during the window is queryable by change number. Supports CAB reviews, SLA reporting, and incident retrospectives without manual reconstruction.

Implementation note

Add change_number as a filterable label in EMS. The existing facet / preset system can then expose a Change preset automatically once the label is present.