Skip to content

EMS Status

This page is a design doc for the current status of EMS (Event Management System) and its components. It includes information on system health, performance metrics, and any ongoing incidents or maintenance activities.
It's intended to provide a quick overview for both EMS owner and users to understand the current state of EMS and identify any potential issues that may require attention.

System Health

Overall system status:

  • Healthy: All components resource usage(LMA metrics & log) are not exceeding thresholds and there are no known issues.
  • Degraded: Some components resource usage(LMA metrics & log) are approaching thresholds, but the system is still operational.
  • Unhealthy: Major components resource usage(LMA metrics & log) are exceeding thresholds or healthcheck fail(MCT) or experiencing significant issues(BUG), impacting system functionality.

Failover status:

  • Normal: No failover events have occurred recently, and all components are functioning as expected.
  • Failover occurred: An automatic/manual failover event was triggered. Show the location of where the service is running now, and the reason for failover if available.

Performance Metrics

Resource Usage

For All Components (Frontend/Backend/DB/Redis/SNMP):

  • CPU Usage: 45% (Healthy)
  • Memory Usage: 60% (Healthy)
  • Disk Usage: 70% (Healthy)
  • Database CPU Usage: 50% (Healthy)

Only give the one not healthy if any component is in degraded or unhealthy status. Else, just show the healthy status for all components.

Alert Volume

  • Alert Received Count: 100,000 alerts/hour (How many alerts system alert reveive endpoint received, include persisted and dropped, should be stable or increasing slightly)
  • Alert Volume: 50,000 alerts/hour (How many alerts are currently being persisted in the system, should be stable or increasing slightly)
  • Alert Processing Queue Size: 500 (How many alerts are waiting in the processing queue, should be not are continue increasing number, The sum of metrics "keep_running_tasks_current" in LMA with 3 backend pods)

User experience

  • API response time: 2s (Average time for EMS API response, should be less than 15s, get the data from LMA metric "keep_http_request_duration_seconds" with 3 backend pods)
  • Workflow execution failure rate: 1% (Percentage of workflow executions that fail, should be less than 5%)
  • DB query duration: 100ms (Average time for database queries to execute, should be less than 500ms)
  • Failure rate of request to backend: 0.5% (The percentage of requests to backend that fail, should be less than 1%)