Skip to content

CUTOVER workaround

Background

MCT is undergoing CUTOVER these days and the migration plan needs to be carried out imperceptibly to ensure that no services are affected. Since there is no proper self check and migrate function in MCT, so we need to find out a way to do this. Here provide a workaround.

Steps

  1. collect all zone_id of timer under specific agent group. Take WBXJFK as example.
  2. query org_id and virtual_group_id of old&new agent from db. 
    select mao.org_name,ma.org_id, ma.virtual_group_id from mct_agent ma
join mct_agent_org mao on mao.org_id = ma.org_id
where mao.org_name = 'wjfkgen-p-1';
  3. run the script below and wait for the result, you need to run it in multiple times, for example, pass in 50 Zone ids each time and wait for the result (otherwise, the server pressure will be too great and affect the existing test). 
    python3 script.py -cec 'chengczh' -zoneids '2360426,2422671,210227,1220076,2060441,960121,960371,2422661,1219866,1220056,194637' -agent 'wjfkgen-p-1'
  4. wait for the script output.

Check script output

    All threads have completed.
    ---------------------------------------------------
    success_zoneids: 123, 321
    fail_zoneids: 456,2360426,2422671
    error_zones(no check under zone): 234192
    time_out_zoneids: [913213]
    offline_zoneids: 987531
    ---------------------------------------------------

    ----------------Step 2 compare history points of each check under failed zones----------------
    ---------------------------------------------------
    below zones get manual test result not same as historical test:
[183132](CSSJ71_SRS):
    [853467][cssj71ott002.webex.com()]: https://mct.prod.webex.com/detail-status/0?isManual=true
    [853462][cssj71ott003.webex.com()]: https://mct.prod.webex.com/detail-status/0?isManual=true
    need to manual check zone ids:                          2360426, 456
    manual test result are same as historical test result:   2422671
Take this as example.

step 1

success_zoneids

This mean the test result are all pass and it is ok to do migration, you can do this in db: 

update mct_timer_setting set org_id = {org_id}, virtual_group_id = {virtual_group_id}
where zone_id in (123,123);

error_zones

This means the corresponding zones don't contain any checks but have timer enabled, we consider this as a waste of timer resources, we need to delete them in mct_timer_setting and add delete action in table mct_change_his). 

-- delete timer if no timer under zone
DELETE FROM mct_timer_setting mts
WHERE mts.zone_id IN (
234192
)
  AND NOT EXISTS (
    SELECT 1 
    FROM mct_server ms 
    WHERE ms.zone_id = mts.zone_id
);

-- add timer delete record in mct_change_his
INSERT INTO mct.mct_change_his (id, table_name, id_name, id_value, data_type, lastmodifiedtime) VALUES (1341074555, 'DEL_MCT_TIMER_SETTING', 'ZONE_ID,TEST_LOCATION,VIRTUAL_GROUP_ID', '234192,GLOBAL_AGENT,1100007703', 2, sysdate());

time_out_zoneids

This means the corresponding zones still in testing status while retry count exceed, need to manual check in MCT manual page.

offline_zoneids

This means the corresponding zones have no manualList from api https://mctapi.prod.webex.com/mpi/test/manual-id/ which is offline.

fail_zoneids

As for result of each check, we consider it as failing if the status is not Pass, such like Fail/No Result/Not Test not_test_case

step 2: analyze fail zones

The fail_zoneids will be analyzed in step 2 of script.

As for fail_zoneids, we need to check if the test results of failed checks under zone in new agent group(wjfkgen-p-1 in this case) are all the same as historical status of old VM(WBXJFK in this case).

manual test result are same as historical test result

If the results are all the same, the conclusion is we can change timer of this zone because the results won't change after timer migration. 

update mct_timer_setting set org_id = 1100098208, virtual_group_id = 1100101302
where zone_id in (2422671);

need to manual check

If the test results are different between old and new agent group, we need to check manually on MCT page. The ACL, app allowlist issue happens in this part. You can query more details to locate these zones: 

select vcz.zone_id, vcz.zone_name, vcz.cluster_name, vcz.cluster_id, mzt.zone_name as zonetype from view_cluster_zone vcz 
join mct_zone mz on mz.zone_id = vcz.zone_id
join mct_zone_template mzt on mzt.zone_type = mz.zone_type
where vcz.zone_id in (
2360426, 456
) order by vcz.cluster_name;

Migration Script

Usage: python3 script.py -cec 'test_user_cec' -zoneids '123,456' -agent 'wdfwgen-p-3'

You could download the script in mct-utility