Skip to content

Architecture in WbxKubed

Architecture Diagram and Data Flow

MCT in WebexKubed platform will be fully in Cloud Native mode, and all the practices are GitOps model, but we also keep the convenience for the operators who are more familiar with UI activities. The service name in WebexKubed are called mas, and MCT's new module name is mas-monitoring. We convert the key components of MCT to micro-services, and cowork with UAS (aka. mas-alerting) to provide a complete Monitoring and Alerting solution.

Architecture Diagram

Components

MAS Central:

  • The Registration Service handles test requests from the MAS Registration Controller in the application cluster or from MAS customer repo hook then aggregate with template do the check registration.
  • The Dispatcher Service manages task dispatch cross all clusters.
  • The Status Service receive test result and store in Pinot, also support forward all test results to ELK (Elasticsearch, Logstash, and Kibana).
  • Pinot retrieves and stores all test results locally.
  • The Alerting Service aggregates all MAS test results and alerts from the Prometheus Alert Manager via webhook. It performs alerting based on tag rules.
  • The Maintain Service receives the aggregated alerts from the Alerting Service and aligns them with all changes from the Change Management System to determine whether do the test or not, whether trigger an alert or not.
  • The Notification Service notifies service owners through multiple channels such as PagerDuty (PD), Webex Teams, email, etc. It also supports ticket creation in the ticketing system. The notification target will render from wbx3-application-group repo.

MAS Workers:

  • MAS worker clusters are created in different regions, with a minimum of 2 clusters required to support cross-region/cluster testing requirements.
  • The MAS Dispatcher Controller receives tasks from the central Dispatcher Service and sends them to Redis.
  • MAS Worker retrieves tasks from Redis, performs the tests, writes the results back to the central Status Service, and also pushes the results as metrics to Prometheus.

Application cluster:

  • In the application cluster, a Controller is placed as a base bundle during cluster build.
  • The MAS Registration Controller detects any new MAS CRs (Custom Resources) created in the cluster and retrieves the relevant information to send to the central system.
  • The MAS Registration CRD (Custom Resource Definition) defines the necessary information so that service owners can create CRs within their application charts.

HA and Reliability Posture

  • Heartbeat Connection: All MAS workers should maintain a heartbeat connection with the MAS central side. This ensures that the central side can make informed decisions for task dispatch based on the workers' availability and status.
  • Cluster Deployment: The MAS central side is deployed in two clusters located in different regions or data centers. One cluster serves as the primary, while the other acts as the backup. This setup enables automatic failover and failback, ensuring high availability and reliability of the MAS system.
  • Metrics and Self-Monitoring: All MAS components should expose relevant application and business metrics to Prometheus, a popular monitoring and alerting system. By leveraging these metrics, MAS performs self-monitoring to proactively detect any anomalies or issues, allowing for prompt remediation and ensuring the overall health and stability of the system.