Cloud Agent Test Flow

This document shows part of the life-cycle of MCT checks, from Timer Setting being Set, to Test Results being Sent Back.

Test Flow ¶

The whole test flow shows in below picture:

Agent / Polling / Plugin / Redis are all MCT backend components.

Process Highlights ¶

Below are some highlights during the test flow, which engineers should pay more attention to.

Agent Leader ¶

Metadata & config data check: only when metadata and config data all finished updating, the process can continue (monitor can not be started when data loading).

Get tasks belong to current minute pos: staled monitors (zoneType < 100) loaded from XML config (MCTConfig.xml); plugin monitors (zoneType > 100) loaded from Redis (DB).

Exclude certain agents.
1. Offline agents from table hash_agent_status (add manually).
```
  ~~~sh
  hget hash_agent_status offline_agents
  ~~~
```
2. Offline agents from table hash_node_status, check current agent's status value, OFFLINE == 2 (agent command SD, and agent self test script fail can both lead to agent's offline).
```
  ~~~sh
  hget hash_node_status:{nodeId} status
  ~~~
```
3. High load agents.
```
  > cpu load >= `0.9` ||  
  > free disk < `500M` ||  
  > free memory < `500M`
```
4. The most busy agent: with the maximum thread count.

Match location rule. Agent is able to isolate some tasks to certain agents, so that some performance issue from part of plugins would not spread to all agent group. format: ZType:{zoneType} = {agentIpList}
```
hgetall hash_mct_location
```

Match isolate rule. Agent is able to isolate some tasks to certain agents, so that some performance issue from part of regions (service / dc) would not spread to all agent group. Format: DC:{dcId}_Service:{serviceId}_Os:{osType} = {agentNodeRegex}
```
hgetall hash_mct_isolation
```

Agent Follower ¶

Need offload for current task: check if zone task matches below conditions.

node load > 0.5 &&
node process > 300 &&
latest zone test fail rate > 50%

Running task count check: > Maximum auto test task count: 200
> Maximum manual test task count: 350

Task timeout: > default 300s for Java Plugin
> default 3600s for Polling

If a task is offloaded, it will be discarded, which will result in missing test results for auto / manual test; If the running task queue is full, no more tasks will be executed, which may result in longer elapsed time for auto / manual test, even timeout.

But such kind of logic is still needed, because it's a self-protection mechanism in case of resource exhaustion in some extreme cases.

Reference ¶

Cloud Agent Test Flow in Miro