MCT Integration with ISS

Background ¶

ISS aggregates data from various monitoring platforms to provide a comprehensive view of Webex services, with MCT being a key data source. This document outlines the design for integrating MCT with ISS.

Requirements Analysis ¶

Determine ISS's needs and what MCT can offer.

Essential Data ¶

• data center
• status_fid / coreFeatureName + domainName
• checkType
• alertId
• link (mct link)
• status (alert / pass)
• message
• alerting check

Integration Triggers with ISS ¶

1. ISS processes notifications tied to Escalation Rules rather than individual alerts.
2. ISS focuses on Core Features, e.g. a single server's health check failure won't trigger an ISS status change, but a VIP's unavailability will.

Strategic Considerations ¶

Ensuring Core Feature Notifications Are Not Suppressed ¶

To minimize noise during service alerts, MCT suppresses notifications when a rule is firing for certain time (default one hour). However, this can cause issues if multiple core features trigger the same alert rule, potentially leading to inaccurate ISS statuses.

Proposed solution: Introduce a core feature flag at the notification level, and MCT notification rule should support suppress by check, instead of suppress by rule, ensuring that critical notifications are not override with each other.

Handling Core Feature Alerts from Different Data Centers ¶

ISS requires visibility into core feature alerts across all data centers for impact analysis. Alerts for the same core feature from different data centers should be treated as separate notifications.

Proposed solution: Create unique checks and alert rules for each core feature - DC pairing.

As a result, relationship for core feature and critical notification is 1 to N; core feature check and critical notification is 1 to 1.

Linking Core Features to ISS ¶

When an alert rule is deemed a critical notification, a link must be established between the notification and ISS. ISS maintains a list of core features with unique IDs - featureId, which must be included in the critical notification settings.

Proposed Solution: Get feature list from ISS, and add a feature select on notification setting page.

Design Action Items ¶

1. Portal: Implement a critical notification checkbox in the alert rule configuration. If selected, display a select for the core feature ID.
2. Backend: When triggering or resolving an escalation rule, verify if it's a critical notification and, if so, invoke the ISS API. (This requires an alert rule recovery feature.)
3. Advanced Features: 1. Preview checks matched by an alert rule during configuration. 2. Add teams notification (to ISS and MCT) when a new core feature is added.

API Definition ¶

When there is a critical notification, MCT will sync the notification to ISS by calling an integration API.

POST /collector/event/v1/receiver?source=mct ¶

Description ¶

Receive core feature notification to from MCT.

Body ¶

{
    "alertId": "28681899-d870-44a4-bb5c-42eb9e61201f",
    "featureId": "status_f88870bd-0973-4d87-b598-4fd092f3649c",
    "timestamp": 1719808956616,
    "checkId": 643747, 
    "checkName": "mttx3mul004.webex.com",
    "checkType": "URL Watch Server",
    "dataCenter": "JFK02_DATA",
    "labels": {
        "mctService": "Monitoring",
        "foo":"bar"
    },
    "mctLink": "http://mct.webex.com/#newDetailStatus.jsp?statusId=1444576901",
    "status": 1, // 0 pass (recover); 1 alert
    "message": "[Start time:] 01:42:25/n[Test results:]/n-----------------------------------------------------------------------------------------/n[Test url:] http://mttx3mul004.webex.com:9876/healthcheck_read.txt/n[Url template:] Template: id=[729201]; url=[http://mttx3mul004.webex.com:9876/healthcheck_read.txt]; post data=[]; port=[] positive/n[mct_healthcheck_ok]; negative=[]/n[IsPass:] false/n[Begin time:] 01:42:25/n[End time:] 01:42:25/n[Positive:] mct_healthcheck_ok/n[Negative:]/n[Detail:] Get \"http://mttx3mul004.webex.com:9876/healthcheck_read.txt\": dial tcp 10.240.56.52:9876: connect: connection refused/n[TrackingID:] MCT_eG4vCZE4xTPK83fLBkoyEg83KXdkhRm3/n[Response content:]/n-----------------------------------------------------------------------------------------/n[Finish time:] 01:42:25"
} 

Note: Since different checks may mapped to same feature, when MCT sending recover events (status == 0), ISS should check if checks under the feature are all OK, before mark this feature is recovered.

Response ¶

• 200 OK

  {
      "errorCode": "OKOKOK",
      "localIp": "10.140.212.59",
      "errorMessage": ""
  }
  

• 400 Bad Request

  {
      "errorCode": "INVALID_INPUT_PARA",
      "localIp": "10.140.212.59",
      "errorMessage": "'checkName' is required."
  }
  

• 500 Server Internal Error

  {
      "errorCode": "NONONO",
      "localIp": "10.140.212.59",
      "errorMessage": "Internal error"
  }