Self Govenance

Overview ¶

MCT is a critical monitoring platform that requires robust self-monitoring to ensure its stability. MCT leverages external tools to monitor its own health and transmits test results to CLP via the EFK stack. Details on this process can be found on the Logs page.

Alert Coverage ¶

Self-monitoring of MCT is validated through OpenSearch. Successful test results indicate the health of the following components:

1. MCT Worker: - Timer functionality - Agent operations (dispatcher / station) - Redis connectivity - Polling processes - Plugin testing
2. MCT Status-API: Reception of test results from workers
3. MCT Central Redis: Transmission of test results to AKKA processor
4. MCT AKKA Processor: Processing of test results
5. EFK Stack: Logging of processed results

Monitoring Types ¶

MCT generates over 20 million test results daily, averaging more than 14,000 results per minute. No data within a recent time frame indicates an anomaly. Monitoring is performed using OpenSearch Alerting, with below two primary monitors set up.

Credentials for these monitors can be found in the Account List.

Monitor: No Status in Recent 1 Minute ¶

• Check Interval: 1 minute
• Data Source: Index metrics-clap_jfk2-cmse-*, Time Field @timestamp
• Query: COUNT of documents in the last 1 minute
• Trigger / Escalation: If count is 0, send email to csg-hz-mct@cisco.com

Monitor: No Status in Recent 5 Minutes ¶

• Check Interval: 1 minutes
• Data Source: Index metrics-clap_jfk2-cmse-*, Time Field @timestamp
• Query: COUNT of documents in the last 5 minutes
• Trigger / Escalation: If count is 0, 1. Trigger PagerDuty for MCT On-Call: send email to ceo-mct-impacthigh@ciscospark.pagerduty.com 2. Update ISS: call ISS webhook

MCT test result in OpenSearch is one of the MCT critical path, so link it with ISS for service health visibility.

Metrics Overview ¶

Below is a visual representation of the metrics monitored: