Retire Service Impact Rule Match Task
Background ¶
Key Service is a MCT concept, that indicate a check is configured with at least one escalation rule. There is a task to timely match the escalation rules and checks, to mark certain checks as key service.
The problem is, the task will only process incremental data, and lots of actions will trigger the task to do re-match, such as check update, rule update, move zone, move cluster etc, the logic is too complicated and problem will occur when task running for certain time.
New Solution ¶
Open source tools like Alert Manager do not need a "match task" for sending alerts, we can also match the rule only when there is an alert comes.
Efficiency should be taken into consideration.
Rule Match Logic Analyze ¶
The original logic of the rule match:
private boolean validateRule(KeyServiceRule rule,KeyServiceServer server){
//check server type
if(!CommonUtil.isEmptyString(rule.getServerType())
&& !rule.getServerType().equals(server.getServerType()) && !rule.getServerType().equals(server.getPriServerType())){
return false;
}
//check hostName
String serverName = server.getHostName();
if(CommonUtil.isEmptyString(serverName)){
serverName = server.getIp();
}
if(CommonUtil.isEmptyString(serverName)){
serverName = "";
}
if (!CommonUtil.isEmptyString(rule.getHostName())){
String[] metacharacterArray = {"*", "+", "|"};
String hostnameRegex = rule.getHostName();
String firstCharacter = hostnameRegex.substring(0,1);
StringBuilder regex = new StringBuilder();
for (String metacharacter : metacharacterArray) {
if (firstCharacter.equalsIgnoreCase(metacharacter)) {
regex.append("[").append(firstCharacter).append("]").append(hostnameRegex.substring(1));
hostnameRegex = regex.toString();
break;
}
}
try {
if (!serverName.matches(hostnameRegex)) {
return false;
}
} catch (Exception e) {
logger.error("updateKeyServiceServer regex match error, regex:"+hostnameRegex+",ruleId:"+rule.getRuleId()+",exception:"+e.toString());
return false;
}
}
//check URL content
String url = server.getUrl();
if(url == null){
url = "";
}
if (!CommonUtil.isEmptyString(rule.getUrlContext())
&& url.trim().indexOf(rule.getUrlContext())==-1){
return false;
}
//finally check VIP
if (rule.getVip() == 1l) {
try {
// TODO need to improve with server info task
if (snowService.queryIsVip(server.getIp())||snowService.queryIsVip(server.getHostName())) {
return true;
}
} catch (Exception e) {
e.printStackTrace();
}
}
return true;
}
If migrating above logic to AutoIncidentTask, a few items need to pay attention to:
!rule.getServerType().equals(server.getPriServerType())can be ignored, since it's only working for check types between 131000 and 132000, which is already retired.- There is a get url logic
String url = server.getUrl();, which needs to interact with DB, so a batch query can be added here. - There is a check VIP logic
rule.getVip() == 1l, need to migrate it to a label based attribute, instead of calling SNOW.
Performance Test ¶
Sample test code:
private void checkKeyService() throws PlatformException {
// get rule list and check list
long startMilli = System.currentTimeMillis();
List<KeyServiceRule> ruleList = keyServiceMapperDao.queryKeyServiceRuleByTime("");
Map<Long, List<KeyServiceRule>> ruleMap = ruleList
.stream()
.collect(Collectors.groupingBy(KeyServiceRule::getServiceId));
List<Long> serviceIds = ruleList.stream().map(KeyServiceRule::getServiceId).collect(Collectors.toList());
Map<Long, List<KeyServiceIgnore>> ignoreList = keyServiceMapperDao.queryKeyServiceIgnoreByTime("")
.stream()
.collect(Collectors.groupingBy(KeyServiceIgnore::getRuleId));
List<Server> servers = serverService.queryAllServers();
long endQueryMilli = System.currentTimeMillis();
// check if rules match
for (Server server : servers) {
List<KeyServiceRule> rules = ruleMap.get(serviceIds.get(new Random().nextInt(serviceIds.size())));
for (KeyServiceRule rule : rules) {
if (isRuleMatch(rule, server)) {
System.out.println("Found Match!!!: " + rule + ", " + server);
}
}
}
long endMilli = System.currentTimeMillis();
System.out.println("rule size: " + ruleList.size() + ", check size: " + servers.size());
System.out.println("total spend: " + (endMilli - startMilli) + "ms");
System.out.println("query spend: " + (endQueryMilli - startMilli) + "ms");
System.out.println("match spend: " + (endMilli - endQueryMilli) + "ms");
System.out.println("========================");
}
private boolean isRuleMatch(KeyServiceRule rule, Server server) {
// skip server type check for now, since rules are imported from Prod, and test env is in qa, so the type is not match
//if(!CommonUtil.isEmptyString(rule.getServerType()) && !rule.getServerType().equals(server.getServerType())){
// return false;
//}
//check hostName
String serverName = server.getServerHostname();
if(CommonUtil.isEmptyString(serverName)){
serverName = server.getServerIp();
}
if(CommonUtil.isEmptyString(serverName)){
serverName = "";
}
if (!CommonUtil.isEmptyString(rule.getHostName())){
String hostnameRegex = rule.getHostName();
String[] metacharacterArray = {"*", "+", "|"};
String hostnameRegex = rule.getHostName();
String firstCharacter = hostnameRegex.substring(0,1);
StringBuilder regex = new StringBuilder();
for (String metacharacter : metacharacterArray) {
if (firstCharacter.equalsIgnoreCase(metacharacter)) {
regex.append("[").append(firstCharacter).append("]").append(hostnameRegex.substring(1));
hostnameRegex = regex.toString();
break;
}
}
try {
if (!serverName.matches(hostnameRegex)) {
return false;
}
} catch (Exception e) {
logger.error("updateKeyServiceServer regex match error, regex:"+hostnameRegex+",ruleId:"+rule.getRuleId()+",exception:"+e.toString());
return false;
}
}
// VIP logic, can be removed later on
/*if (rule.getVip() == 1l) {
try {
// TODO need to improve with server info task
if (snowService.queryIsVip(server.getIp())||snowService.queryIsVip(server.getHostName())) {
return true;
}
} catch (Exception e) {
e.printStackTrace();
}
}*/
return true;
}
In above benchmark example, only cares about hostname regex match, since it's the most time-consuming part. Below is the result of 6 times:
rule size: 449, check size: 23570
total spend: 16317ms
query spend: 1130ms
match spend: 15168ms
rule size: 449, check size: 23570
total spend: 16048ms
query spend: 737ms
match spend: 15304ms
rule size: 449, check size: 23570
total spend: 14169ms
query spend: 512ms
match spend: 13652ms
rule size: 449, check size: 23570
total spend: 12984ms
query spend: 531ms
match spend: 12444ms
rule size: 449, check size: 23570
total spend: 12897ms
query spend: 598ms
match spend: 12294ms
rule size: 449, check size: 23570
total spend: 14067ms
query spend: 578ms
match spend: 13483ms
For 23,570 checks, max time spend on rule match is 15304ms, min time is 12,294ms, avg time is 13,724ms. For single check, max time on rule match is 0.64ms, which is less than a redis interaction cycle. In real cases, since there would be more filter labels in a rule, so the time cost should be much lower.
Max list size on Production ¶
Check ids need to be processed is in a redis list. Example code:
max_length=0
while true; do
current_length=$(redis-cli -h 10.254.173.49 -p 7002 -a xxx -c LLEN list_waiting_incident_server_id 2>/dev/null)
if (( current_length > max_length )); then
max_length=$current_length
fi
echo "Current length: $current_length, Max length: $max_length"
sleep 1
done
Latest result in past 2 hour:
Current length: 57, Max length: 291
Which means in real cases, the max total time spend for Prod checks rule match is 114ms, we can handle it easily.
Rule Matched Check Design ¶
In previous logic, there is a matched rule set on check level, to maintain those rules matching current check, so if want to know how many checks do a rule matches, just need to query check list from db, and filter by ruleId. But now we need to calculate the matched checks in real time, luckily PostgreSQL support regex match, so just one DB query is enough to find the rule matched checks.
Below is a SQL example with complicated regex expression:
select * from mct_server where server_hostname ~ '(idpush|fls|idchglog)(-ta|-sj)(-001-bts|-002-bts|-003-bts|-004-bts|-005-bts|-006-bts|-007-bts|-008-bts).amer.prv.webex.com|y90(tx|sj)1y(ib|cn)0(01|02|03|04|05|06|07|08|09|10).webex.com|(ciflsbts-us|ciflsbts|idbrokerbts|identitybts|ciflsbts-ta-s|idbrokertxbts-s|identitytxbts-s|idpushtxbts-s|cloudconnectorsjbts-s.cisco.comidbrokersjbts-s|identitysjbts-s|idpushsjbts-s).webex.com|(cloudconnectorbts|cloudconnectortxbts-s|cloudconnectorsjbts-s).cisco.com';
and the result as below:
[2024-01-26 16:28:37] 11 rows retrieved starting from 1 in 1 s 367 ms (execution: 1 s 118 ms, fetching: 249 ms)
pggmct: mct, public, pg_catalog> select * from mct_server where server_hostname ~ '(idpush|fls|idchglog)(-ta|-sj)(-001-bts|-002-bts|-003-bts|-004-bts|-005-bts|-006-bts|-007-bts|-008-bts).amer.prv.webex.com|y90(tx|sj)1y(ib|cn)0(01|02|03|04|05|06|07|08|09|10).webex.com|(ciflsbts-us|ciflsbts|idbrokerbts|identitybts|ciflsbts-ta-s|idbrokertxbts-s|identitytxbts-s|idpushtxbts-s|cloudconnectorsjbts-s.cisco.comidbrokersjbts-s|identitysjbts-s|idpushsjbts-s).webex.com|(cloudconnectorbts|cloudconnectortxbts-s|cloudconnectorsjbts-s).cisco.com'
[2024-01-26 16:57:14] 11 rows retrieved starting from 1 in 1 s 175 ms (execution: 1 s 116 ms, fetching: 59 ms)
Time cost is about 1.2s, which is acceptable.
Conclusion ¶
Calculate the check matched rules, and rule matched checks are all doable.