Skip to content

AI Analysis Trigger Design Document

1. Overview

This document outlines how EMS automatically triggering an AI analysis agent in response to specific alerts and incidents. The goal is to provide timely, automated analysis for potentially critical events through a trigger rules with matcher, Alert Flaping Suppression and Event Grouping.

2. Core Concepts

  • Trigger Rule: A configuration object that defines the conditions under when and which AI agent should be called.
  • Matcher (CEL): A Common Expression Language (CEL) query within a rule that evaluates the alert event or incident's data.
  • Event Grouping: A mechanism to group related events (e.g., multiple alerts for the same service) to ensure only one analysis is performed for the entire group.
  • Alert Flaping Suppression: A process to prevent re-analyzing the same alerts if it occurs multiple times in a short period.
  • Throttling: A rate-limiting mechanism to control the frequency of agent calls, preventing system overload and unnecessary cost.

3. Workflow

The process from event to AI analysis follows these steps:

  1. Event Ingestion: An alert is received (e.g., status is firing) or an incident is created (status is firing).
  2. Rule Evaluation: The system iterates through all active Trigger Rules and evaluates the event against each rule's CEL matcher.
  3. Event Grouping & Alert Flaping Suppression: If a rule matches, use have set the grouping key in the rule and duplicate fingerprint already in the event. - If the event can be grouped into an existing event group (based on the grouping key and time window), it is added to that group. - If the event belongs to an already-analyzed event group, the trigger is suppressed. - If an identical alert event fingerprint(without timestamp postfix) or incident id has been seen recently, the trigger is suppressed.
  4. Throttling Check: The system checks if triggering the agent would violate any rate limits (e.g., minimum interval between calls for the same rule/group). If so, the trigger is suppressed.
  5. Agent Dispatch: If all checks pass, call the specified AI analysis agent with the event's context.
  6. Result: Wait for AI return the analysis result, if the response is over certain time, mark as inprogress and wait agent call back with the result.
sequenceDiagram 
    participant E as Event (Alert/Incident)
    participant R as Rule Engine
    participant C as Grouping/Suppression
    participant T as Throttling
    participant A as AI Agent

    E->>R: Ingest Event
    R->>R: Evaluate Rules (CEL Matcher)
    alt Rule Matches
        R->>C: Check Grouping & Suppression
        alt Not Suppressed
            C->>T: Check Throttling
            alt Not Suppressed
                T->>A: Call AI Agent with Event Context
                A-->>R: Return Analysis Result
            else Suppressed
                T-->>R: Suppress Trigger (Throttling)
            end
        else Suppressed
            C-->>R: Suppress Trigger (Grouping/Suppression)
        end
    else No Match
        R-->>E: No Action
    end

Sample Scenario

Imagine a database service db-prod starts to fail. Set grouping key to db-latency:db-prod with a 15-minute window, duplicate suppression TTL to 10 minutes, and minInterval to 1 minute.

Timeline of Events:

  1. 10:00 AM: Alert HighLatency fires for db-prod. - Rule Matches: The rule for DB issues is a match. - Event Grouping: A new group is created with key db-latency:db-prod. - Alert Flaping Suppression: This is a new event fingerprint. - Throttling: No recent triggers. - Result: Agent is called.

  2. 10:01 AM: Alert HighErrorRate fires for db-prod. - Rule Matches: Yes. - Event Grouping: The key is db-latency:db-prod, which matches the group created at 10:00 AM. The event is added to the group. - Result: Trigger is suppressed (by Event Grouping). The system assumes the first analysis will cover this new symptom.

  3. 10:05 AM: The HighLatency alert from 10:00 AM resolves and immediately fires again (a "flap"). - Rule Matches: Yes. - Event Grouping: The key matches the existing group. - Alert Flaping Suppression: The event fingerprint is identical to the one from 10:00 AM and it's within the TTL. - Result: Trigger is suppressed (by Alert Flaping Suppression).

  4. 10:45 AM: A completely new issue occurs, and HighLatency fires again for db-prod. - Rule Matches: Yes. - Event Grouping: The original group from 10:00 AM is now outside its 15-minute window, so a new group is created. - Alert Flaping Suppression: The fingerprint is the same, but it's outside the TTL. - Throttling: The minIntervalPerKey is 1 minutes. The last trigger for this key was at 10:00 AM, which was 45 minutes ago. The check passes. - Result: Agent is called.

4. Rule Definition

Trigger Type

  • Event Type: Alert or Incident

Matcher

  • CEL expression

Event Grouping & Alert Flaping Suppression

  • grouping key: a template string to define how to group related events.
  • grouping window: the time period to group related events.
  • alert flaping suppression ttl: the time period to ignore identical events base on fingerprint.

Throttling

  • minInterval: The minimum interval between calls for the same rule.

Action

  • Agent Selection: Select which AI agent to call.
  • Timeout: Maximum wait time for the agent call back before marking the analysis as in-progress.
  • Payload: Define what data from the event should be sent to the agent. By default, send the whole event body data.（Phase 2）

5. Agent Interaction

  • By default, EMS integrated with PCCAgent, will call the agent API with the event context, Authentication uses existing CI solution.
  • By default, The auto trigger analysis call PCCAgent will not wait for response，need agent call back with the result.
    post body: 
    {
    "sessionId": "{{sessionId}}",
    "chatInput": "{{alert or incident body}}", 
    ## will enrich 3 fields into origin alert / incident body: "AlertFrom":"EMS","callback_path":"xxxxx","need_callback":"true" 
}
  • EMS provides a API to let AI agent send back the analysis result.
    post /ai/agent/callback 
    {
    "sessionId": "{{sessionId}}",
    "result": "{{callback result}}",
    "endTime": "{{ai analysis end time}}",
    "executionTime": "{{execution duration with seconds}}"
}
  • For future, we may integration with more agent then we can select the agent when define the rule.