AI Analysis Result Feedback Collection System ¶

1. Overview ¶

This document outlines the design for collecting user feedback on AI analysis results in the EMS system. The goal is to understand how AI-generated root cause analysis helps users resolve incidents and continuously improve the AI model's accuracy and usefulness through both active and passive feedback mechanisms.

2. Core Concepts ¶

• Active Feedback: Explicit user actions that indicate satisfaction or dissatisfaction with AI analysis results
• Passive Feedback: Implicit user behaviors that can be tracked to infer the usefulness of AI analysis

3. Feedback Collection Strategy ¶

3.1 Active Feedback Mechanisms ¶

3.1.1 Immediate Feedback (Analysis Result Page) ¶

Location: Displayed prominently on the AI analysis result tab/page.
Behavior and Interaction:
1. Minimal Initial Ask(required): Start with thumbs up/down with "Help us improve!" prompt.
2. Smart Second Layer Feedback(optional):

**If thumbs down feedback, immediately ask "What went wrong?"**    
- ❌ Incorrect root cause    
- ❌ Missing important information    
- ❌ Irrelevant suggestions

**If thumbs up feedback, ask "What worked best?"**     
- ✅ Accurate root cause identified    
- ✅ Helpful troubleshooting steps    
- ✅ Helpful remediation suggestions     
- ✅ Resolve alert/incident

1. Detailed Feedback Option(optional): Expandable section for free text comments

Timing and Contextual Prompts:
1. Show at the Beginning/End: Show the "Help us improve!" feedback widget when user first lands on the analysis page.
2. Contextual Timing:
Minimum Feedback Threshold: 20 feedback submissions per AI runbook

3.2 Passive Feedback Mechanisms ¶

3.2.1 User Behavior Tracking ¶

Page Engagement Metrics:
- Time spent on AI analysis page
- Scroll depth and reading patterns
- Copy/paste actions from analysis content
- Click the links provided in the analysis report

4. Feedback Data Model ¶

4.1 Active Feedback Schema ¶

Feedback:

{
    "ai_runbook_id": "uuid",
    "feedback_id": "uuid",
    "analysis_id": "uuid",
    "alert_id": "uuid",
    "user_id": "string",
    "feedback_type": "immediate|followup|survey",
    "timestamp": "timestamp",
    "satisfaction": "good|bad",
    "categories": [
        "incorrect_root_cause",
        "missing_information",
        "irrelevant_suggestions",
        "accurate_root_cause",
        "helpful_troubleshooting",
        "helpful_remediation",
        "resolved_incident"
    ],
    "binary_helpful": true,
    "free_text": "string"
}

4.2 Passive Feedback Schema ¶

• Time spent on AI analysis page
• Scroll depth and reading patterns
• Copy/paste actions from analysis content
• Click the links provided in the analysis report

{
    "session_id": "uuid",
    "analysis_id": "uuid",
    "user_id": "string",
    "engagement_metrics": {
        "view_duration_seconds": 0,
        "scroll_percentage": 0,
        "clicks_on_suggestions": 0,
        "copy_paste_actions": 0
    }
}

5. Report Scoring System ¶

5.1 Score Display ¶

Location: Displayed prominently at the top of every AI analysis report.

Visual Representation: 1. Show score in the analysis report header:

┌─────────────────────────────────────────────────────────────┐
│  AI Analysis Report                                         │
│  ┌─────────────────────────────────────────────────────┐   │
│  │ 🎯 Reliability Score: 8.5/10  (Based on 47 votes)   │   │
│  │ ████████████████████░░░░  HIGH CONFIDENCE           │   │
│  └─────────────────────────────────────────────────────┘   │
│                                                             │
│  Root Cause Analysis:                                       │
│  ...                                                        │
└─────────────────────────────────────────────────────────────┘

Score Tiers:

1. Show different colors in alert AI analysis icon with different scores

5.2 AI Runbook Binding ¶

Each AI analysis report is bound to an ai_runbook identifier in the alert body: Score Model:

{
    "ai_runbook_id": "uuid",
    "ai_runbook": "string",
    "score": 9.0,
    "feedback_count": 12
}

6. Score Calculation Algorithm ¶

6.1 Scoring Components ¶

The score is calculated using a weighted combination of Active and Passive feedback:

6.2 Active Feedback Scoring (90%) ¶

Phase 1: Thumbs Up/Down with Confidence Weighting

Simple thumbs up/down ratio adjusted by sample size confidence:

Base Score = (Thumbs Up Count / Total Feedback Count) × 10
Confidence Factor = min(Total Feedback Count / 20, 1.0)
Active Score = Base Score × Confidence Factor + 5.0 × (1 - Confidence Factor)

Where:
- 👍 Thumbs Up: counts as 1
- 👎 Thumbs Down: counts as 0
- Default Score: 5.0 (neutral starting point)
- Confidence Threshold: 20 feedbacks

Example Calculations:

Why Confidence Weighting? - Prevents extreme scores from small sample sizes (e.g., 1 thumbs up = 10.0) - New runbooks start near neutral (5.0) until enough feedback is collected - Score becomes more "real" as feedback count increases

Phase 2 (Future): Include Sub-item Weights

Sub-items can be added later to provide more granular scoring: - ✅ Accurate root cause: +0.3 - ✅ Helpful troubleshooting: +0.2 - ✅ Helpful remediation: +0.2 - ✅ Resolved incident: +0.5 - ❌ Incorrect root cause: -0.3 - ❌ Missing information: -0.2 - ❌ Irrelevant suggestions: -0.2

6.3 Passive Feedback Scoring (10%) ¶

Passive Score = (Engagement Score) * 10

Engagement Score Components:
- View Duration: 
  - < 10 seconds: 0.0 (too short, likely not read)
  - 10-30 seconds: 0.3
  - 30-120 seconds: 0.6 (optimal reading time)
  - > 120 seconds: 0.4 (may indicate confusion)

- Scroll Percentage:
  - 0-25%: 0.0
  - 25-50%: 0.1
  - 50-75%: 0.2
  - 75-100%: 0.3

- Copy/Paste Actions:
  - 0: 0.0
  - 1-2: 0.2
  - 3+: 0.3 (strong signal of usefulness)

- Link Clicks:
  - 0: 0.0
  - 1+: 0.1

Maximum Passive Score: 1.0 → Normalized to 10

6.4 Final Score Calculation ¶

Final Score = (Active Score * 0.9) + (Passive Score * 0.1)

Example:
- Active Score: 8.0 (mostly positive feedback)
- Passive Score: 7.0 (good engagement)
- Final Score: (8.0 * 0.9) + (7.0 * 0.1) = 7.2 + 0.7 = 7.9

6.5 Score Decay and Recency Weighting ¶

To ensure scores reflect recent performance, apply time-based weighting:

Time Weight = e^(-days_ago / 90)

Where:
- Feedback from today: weight = 1.0
- Feedback from 30 days ago: weight ≈ 0.72
- Feedback from 90 days ago: weight ≈ 0.37
- Feedback from 180 days ago: weight ≈ 0.14

This value can be adjusted based on business needs (30 days for more aggressive decay, 180 days for more conservative)

Example Calculation:

Assume an AI Runbook has 4 feedback submissions:

Weighted Score Sum = 9.0 + 5.76 + 1.48 + 0.42 = 16.66
Weight Sum = 1.0 + 0.72 + 0.37 + 0.14 = 2.23

Final Weighted Score = 16.66 / 2.23 = 7.47

Comparison with Simple Average:

Simple Average = (9.0 + 8.0 + 4.0 + 3.0) / 4 = 6.0
Weighted Average = 7.47

The time-weighted approach better reflects recent improvements, encouraging continuous enhancement of AI analysis quality.

7. Analytics and Reporting ¶

7.1 Feedback Metrics Dashboard ¶

Key Performance Indicators:
- Helpfulness Score: Average rating across all feedback
- Feedback Volume: Number of feedback submissions per day/week
- Resolution Impact: Percentage of users who report AI analysis helped resolve
- Time Savings: Average time saved when AI analysis is used
- Accuracy Trends: Improvement in accuracy ratings over time

Feedback Distribution:
- Positive vs. negative feedback ratios
- Most common feedback categories
- User segments with highest/lowest satisfaction
- Most/least useful for different service/subagent

8. Related Features Changes ¶

• Auto add field ai_report_url to incident/alert detail body when EMS AI analysis is triggered.
• Add field ai_report_url to SNOW Issue creation workflow so that SNOW can link back to the AI analysis report.
• Add feedback widget to manual resolve incident/alert dialog if the incident/alert has AI analysis report.