EMS Operation Runbook ¶

1. Running status check ¶

Check HA status ¶

Visit healthcheck detail /v2/healthcheck/ha/detail check which side is running on right now then check each side healthcheck status, normally primary side will be OKOKOK and backup side will be NONONO.

Componment running status check ¶

Check the EMS componment running status, including frontend/backend/DB/redis/SNMP, from LAM dashboard: EMS Resource Usage Dashboards

See any instance number of each components reporting metrics is smaller than normal, which may indicate there is issue for that component. For example, if only 2 of 3 backend instances are reporting metrics, it may indicate there is issue for the other backend instance which is not reporting metrics.

2. Performance degradation troubleshooting ¶

When there is performance degradation issue for EMS, like we got PD of auto failover happened, the following steps can be taken to troubleshoot the issue:

Resources usage check ¶

Check the resource usage of EMS components, including CPU, memory, disk for all components(frontend/backend/DB/redis) from LAM dashboard: EMS Resource Usage Dashboards

See any abnormal resource usage, especially for DB CPU usage, which is the most common issue for EMS performance degradation.

Incoming alerts check ¶

Check the incoming alert volume by LMA dashboard: EMS Incoming Alerts Dashboards

• Look at the panel "Daily Total InComing Alerts Count" see if there is a sudden increase of incoming alert volume, which may cause performance degradation.
• Look at the panel "Alerts Count by Service" and "Alerts Count by Provider" see if there is any abnormal increase for a specific service or provider.
• Look at the panel under row "OCP & kubed & Network" and "Wx CCE/DI" see more detail. If any count is more than 100,000 for 1 days, it's considered as high alert volume, which may cause performance degradation.
• Look at the panel "Drop History" to see if there is any alert drop count is increasing, which indicates the system is overwhelmed by incoming alert volume.

Check the incoming alert volume by LMA dashboard: EMS main focus Dashboards

• Look at the panels "Alerts Count from XXX" and check which alerts contribute the most incoming alert volume in last 24 hours, which may cause performance degradation.

Log check ¶

Check if there is any abormal logs in OpenSearch

Add tages:ems2-backend and datacenter:wjfkgen-p-1(the kube cluster name), and search for log message: "xxx"(the log key word), to check if there is anything wrong.

Some key word like:

• "Fetched alerts from DB" see if there is abnormal big number for fetched alerts.
• "Error" or "Exception" overall to see if there is any error log, and check the error details.
• "Alert insert queue size" to see if the alert insert queue is remaining big size, which indicates the system is overwhelmed.
• "DB duration" if there is any big query cause more than 30s to get the result, check the query and investigate why it's slow, where trigger the query.

Trace check ¶

Todo