Skip to content

Auto Failover For EMS

Overview

This document describes the auto failover mechanism for EMS composed of two main components: frontend and backend. The backend connects to a PostgreSQL database and Redis. The service is deployed across two data centers (primary and secondary) to ensure high availability and resilience.

Architecture

  • Components:
    • Frontend
    • Backend (connects to PostgreSQL DB and Redis)
  • Deployment:
    • Two data centers: Primary and Secondary
    • Global DNS manages traffic routing based on health checks

Health Check Mechanism

  • Global DNS:
    • Monitors health of both data centers
    • By default, routes traffic to primary
    • Routes traffic to secondary if primary fails health check
  • Frontend Health Check:
    • Call backend health check returns fail if backend health check fails
  • Backend Health Check:
    • Verifies DB connectivity, query the alert table with limit 1
    • Retrieves HA flag active (0 or 1) from Redis
    • Success if flag is 1 and DB/Redis are reachable
    • Fails if flag is 0 or DB/Redis are unreachable
    • Retries DB/Redis check up to 3 times before failing

HA Flag and Failover APIs

  • HA Flag (active):
    • active: 1 or 0 in redis
    • Noramlly only one side can have active: 1, only if two side network broken not connected able to each other
  • API Endpoints:
    • GET /v2/healthcheck/ha: Retrieve current HA flag value
    • POST /v2/healthcheck/ha: Set HA flag value (0 or 1)

Manual and Automatic Failover

  • Manual Failover/Failback:
    • Use POST /v2/healthcheck/ha/failover to switch current active side to inactive side, validates peer status before change which means the other side must be healthy
    • Then the automatic failover logic on the other side will set itself to active:1 after confirming the oppsite is down

  • Automatic Failover Logic:

    • Primary:
    • Check itself health every minute
    • If db or redis error, set itself as active:0 （this is for make primary side health check fail and let GSB side take over）
    • If self health return NONONO, check other side health
    • If GSB side health return OKOKOK, then do nothing
    • If GSB side health return NONONO after retries 2 times with 6 seconds interval, set self as active: 1 in Redis and send failback PD, which means GSB down, primary side take over (auto force failback)
    • GSB:
    • Check Primary health every minute
    • If return OKOKOK, set itself as active:0
    • If return NONONO, retries 2 times, then set itself as active:1 and send failover PD, which means Primary down, Secondary take over (auto failover)
  • Critical processes(e.g., consumer providers):
    • Only run on the active side which means the side frontend healthcheck is ok

Sequence Diagrams

Health Check Flow

sequenceDiagram
    participant User
    participant DNS
    participant Frontend
    participant Backend
    participant DB
    participant Redis

    User->>DNS: Request Service
    DNS->>Frontend: Route to healthy datacenter
    Frontend->>Backend: Health Check
    Backend->>DB: Check DB connectivity (3 retries)
    Backend->>Redis: Get HA flag "active"
    alt HA flag = 1 and DB/Redis OK
        Backend->>Frontend: Health OK (OKOKOK)
        Frontend->>DNS: Health OK
    else HA flag = 0 or DB/Redis fail
        Backend->>Frontend: Health Fail (NONONO)
        Frontend->>DNS: Health Fail
        DNS->>Frontend: Route to secondary
    end

Auto Failover Flow (GSB monitors Primary)

sequenceDiagram
    participant GSB as GSB (Secondary)
    participant Primary
    participant Redis as GSB Redis
    participant PD as PagerDuty

    loop Every 1 minute
        GSB->>Primary: Check Primary health
        alt Primary returns OKOKOK
            GSB->>Redis: Set active = 0
            Note over GSB: Stay inactive
        else Primary returns NONONO
            GSB->>Primary: Retry #1 (wait 6s)
            GSB->>Primary: Retry #2 (wait 6s)
            alt Still NONONO after retries
                GSB->>Redis: Set active = 1
                GSB->>PD: Send Failover Alert
                Note over GSB: GSB takes over traffic
            end
        end
    end

Auto Failback Flow (Primary self-recovery)

sequenceDiagram
    participant Primary
    participant GSB as GSB (Secondary)
    participant Redis as Primary Redis
    participant PD as PagerDuty

    loop Every 1 minute
        Primary->>Primary: Check self health (DB/Redis)
        alt DB or Redis error
            Primary->>Redis: Set active = 0
            Note over Primary: Let GSB take over
        else Self health returns NONONO
            Primary->>GSB: Check GSB health
            alt GSB returns OKOKOK
                Note over Primary: Do nothing, GSB is handling traffic
            else GSB returns NONONO
                Primary->>GSB: Retry #1 (wait 6s)
                Primary->>GSB: Retry #2 (wait 6s)
                alt Still NONONO after retries
                    Primary->>Redis: Set active = 1
                    Primary->>PD: Send Failback Alert
                    Note over Primary: Primary force takes over (GSB down)
                end
            end
        end
    end

Manual Failover Flow

sequenceDiagram
    participant User
    participant Active as Active Side
    participant Inactive as Inactive Side
    participant Redis

    User->>Active: POST /v2/healthcheck/ha/failover
    Active->>Inactive: Check peer health
    alt Peer is healthy (OKOKOK)
        Active->>Redis: Set self active = 0
        Note over Active: Health check now returns NONONO
        Note over Inactive: Auto failover logic detects Active is down
        Inactive->>Redis: Set self active = 1
        Note over Inactive: Traffic switches to Inactive side
    else Peer is unhealthy (NONONO)
        Active->>User: Reject failover (peer not ready)
    end

Failure Scenarios

  • Primary Datacenter Down:
    • DNS detects failure, routes traffic to secondary
    • Secondary sets active: 1 after confirming primary is down
  • Split-Brain Prevention:
    • Manual and automatic failover always check peer's health and HA flag before switching

API Reference

  • GET /v2/healthcheck/ha:
    • Returns current HA flag value
  • POST /v2/healthcheck/ha (body: { "active": "0" | "1" }):
    • Sets HA flag value
    • Validates peer status before change
  • Get /v2/healthcheck/ha/detail:
    • Returns current endpoint healthcheck details
  • POST /v2/healthcheck/ha/failover:
    • Manual failover, set self active as 0, set remote active as 1