Skip to content

High Availability

High Availability

image-20231207165707412

VIP Configuration

The DNS for both uas.prod.meetapi.webex.com and uas-ext.prod.meetapi.webex.com is configured via Route 53. Their health checks are set up based on the externally exposed APIs: https://mas-alert-portal.ext.wdfwgen-p-3.prod.infra.webex.com/notification/healthCheck and https://mas-alert-portal.ext.wjfkgen-p-1.prod.infra.webex.com/notification/healthCheck, respectively.

The liveness probe, which is configured in the notification service, monitors the health of all other services and dependent components. If any of these components fail or go down, the notification service will keep restarting. As a result, the Virtual IP (VIP) on that side would also fail.

Internal

The website uas.webex.com is mapped to uas.prod.meetapi.webex.com. This, in turn, points to two other sites. The secondary (SEC) one is mas-alert-portal.int.wdfwgen-p-3.prod.infra.webex.com and the primary (PRI) one is mas-alert-portal.int.wjfkgen-p-1.prod.infra.webex.com.

External

The website uas.ext.webex.com is linked to uas-ext.prod.meetapi.webex.com. This subsequently directs to two other sites. The secondary (SEC) site is mas-alert-portal.ext.wdfwgen-p-3.prod.infra.webex.com and the primary (PRI) site is mas-alert-portal.ext.wjfkgen-p-1.prod.infra.webex.com.

Process of Failover/Failback

Route53 evaluates the health check of the notification service and determines which side to failover to.

Manual Failover Procedure

  • An API request initiates the failover process. If the UAS is already on the secondary side, no action is taken, and the API returns "failovered".
  • If the UAS is on the primary side, the backend directs the request to the secondary side. At this point, the current time is t1, active region is on the secondary side, it begins clearing previous cache data pre-loading cache data from Pinot to REDIS for the last three hours (t1-3h, t1), without consuming any new alerts. Then the API returns "failovering".
  • Once the secondary side finishes pre-loading data from t1-3h to t1, the health status of the primary side changes to 500, meanwhile, make sure that the health status of the secondary side is 200,
  • The scheduler on the secondary side checks the cluster status regularly. If the status changes from inactive to active, it continues pre-loading all the remaining Pinot data (t1, now), and after completing this, the API returns "failovered".

Manual Failback Procedure

  • An API request initiates the failback process. If the UAS is already on the primary side, no action is taken, and the API returns "failbacked".
  • If the UAS is on the secondary side, the backend directs the request to the primary side. At this point, the current time is t2, active region is on the primary side, it begins clearing previous cache data and pre-loading cache data from Pinot to REDIS for the last three hours (t2-3h, t2), without consuming any new alerts. Then the API returns "failbacking".
  • Once the secondary side finishes pre-loading data from t2-3h to t2, the health status of the secondary side changes to 500, meanwhile, make sure that the health status of the primary side is 200,
  • The scheduler on the primary side checks the cluster status regularly. If the status changes from inactive to active, it continues pre-loading all the remaining Pinot data (t2, now), and after completing this, the API returns "failbacked".

Automatic Failover Procedure

  • The scheduler on the secondary side regularly checks the cluster status. If it changes from inactive to active, the scheduler begins clearing previous cache data and pre-loading cache data from Pinot to REDIS for the previous hour, without consuming any new alerts.
  • Once the pre-loading on the secondary side is complete, it resumes consuming alerts.

Automatic Failback Procedure

  • The scheduler on the primary side regularly checks the cluster status. If it changes from inactive to active, the scheduler begins pre-loading cache data from Pinot to REDIS for the previous hour, without consuming any new alerts.
  • Once the pre-loading on the primary side is complete, it resumes consuming alerts.