Resource Manager Refactor

This doc is about the enhancement and refactor of Resource Manager, to improve usability and facilitate future maintenance.

Key Points ¶

Below are the key points that the refactor should take into concern.

Deploy Resource Manager on Central instead of each agent.
Split Machine resource and Account resource.

Overall architecture ¶

Account Management ¶

Account Table DDL ¶

-- pri
create sequence s_mct_resource_account
    minvalue 1000000001
    increment by 2
    maxvalue 9999999999
    cache 10
    cycle;
-- gsb
create sequence s_mct_resource_account
    minvalue 1000000000
    increment by 2
    maxvalue 9999999999
    cache 10
    cycle;

create table mct_resource_accounts
(
    id                 bigint default nextval('s_mct_resource_account'::regclass) not null primary key, -- the id
    test_app           varchar(255)                                           not null,             -- test type, meeting / outlook...
    app_urls           varchar(255) default '[]',                                                   -- the app url list, should be a json array !!!UPDATE!!!
    app_name           varchar(255)                                           not null,             -- app name / site name !!!NEW!!!
    username           varchar(255)                                           not null,
    password           varchar(255)                                           not null,
    bound_machine_ip   varchar(255),
    status             integer default 3,                                                           -- 0: busy; 1: ready
    requester_id       varchar(255),                                                                -- requester ip+uuid
    release_time       timestamp default CURRENT_TIMESTAMP,                                         -- release timestamp !!!NEW!!!
    created_time       timestamp default CURRENT_TIMESTAMP,
    last_modified_time timestamp default CURRENT_TIMESTAMP
);

CREATE OR REPLACE FUNCTION mct.trigger_fct_tr_mct_resource_accounts_update()
 RETURNS trigger
 LANGUAGE plpgsql
 SECURITY DEFINER
AS $function$
BEGIN
    NEW.lastmodifiedtime := sysdate();
    RETURN NEW;
END;
$function$
;
CREATE TRIGGER tr_mct_resource_accounts_update
BEFORE INSERT OR UPDATE ON mct_resource_accounts
FOR EACH ROW
EXECUTE FUNCTION trigger_fct_tr_mct_resource_accounts_update();

Create Account ¶

Admin can add the account through MCT portal

Path: /management/test-accounts
Method: POST

Body Example:

{
    "testApp": "meeting",
    "appUrls": [
        "mct-ci-testing.webex.com"
    ],
    "appName": "mct-ci-testing",
    "username": "mctuser14@yopmail.net",
    "password": "xxx",
    "boundMachineIp": "10.224.251.200"
}

- Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

Update Account ¶

Admin can update the account through MCT portal

Path: /management/test-accounts/{id}
Method: PUT

Body Example:

{
    "testApp": "meeting",
    "appUrls": [
        "mct-ci-testing.webex.com"
    ],
    "appName": "mct-ci-testing",
    "username": "mctuser14@yopmail.net",
    "password": "xxx",
    "boundMachineIp": "10.224.251.200"
}

- Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

Get Account List ¶

Get the account list, if appUrl is not provided, will return the whole account list

Path: /management/test-accounts?appUrl={appUrl}
Method: GET

Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "172.16.3.18",
    "result": [
        {
            "status": 3,
            "requesterInfos": [],
            "releaseTime": "2025-01-24T14:58:46.231648",
            "id": 0,
            "testApp": "meeting",
            "appUrls": [
                "https://go.webex.com"
            ],
            "appName": "go",
            "username": "mct-test",
            "password": "xxx",
            "boundMachineIp": "127.0.0.1",
            "createdTime": "2025-01-23T05:38:58.971107",
            "lastModifiedTime": "2025-02-10T08:14:40",
            "ready": true,
            "enable": true
        },
        {
            "status": 3,
            "requesterInfos": [
              {
                  "id": 999,
                  "resourceId": 333,
                  "resourceType": "ACCOUNT",
                  "requesterId": "111",
                  "assignTime": "2025-04-25T03:12:15.339178",
                  "timeoutTime": "2025-04-25T03:12:15.339178"
              }
            ],
            "releaseTime": "2025-02-12T02:57:56",
            "id": 1000000361,
            "testApp": "meeting",
            "appUrls": [
                "https://dummy.webex.com"
            ],
            "appName": "dummy",
            "username": "zhochi1",
            "password": "xxx",
            "boundMachineIp": null,
            "createdTime": "2025-01-26T16:47:59.847571",
            "lastModifiedTime": "2025-02-12T02:57:56",
            "ready": true,
            "enable": true
        },
    ]
}

Delete Account By ID ¶

Delete account by id

Path: /management/test-accounts/{id}
Method: DELETE

Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

Windows Machine Management ¶

Since resource is managed on central, need a namespace to distinguish them. Similar with agent, we can use kubed-cluster name or pool name as a namespace.

Machine Table DDL ¶

-- pri
create sequence s_mct_resource_machine
    minvalue 1000000001
    increment by 2
    maxvalue 9999999999
    cache 10
    cycle;
-- gsb
create sequence s_mct_resource_machine
    minvalue 1000000000
    increment by 2
    maxvalue 9999999999
    cache 10
    cycle;

create table mct_resource_machines
(
    id                 bigint default nextval('s_mct_resource_machine'::regclass) not null primary key, -- the id
    ip                 varchar(255)                                           not null,
    hostname           varchar(255)                                           not null,
    port               integer                                                not null,
    username           varchar(255)                                           not null,
    status             integer default 3073,                                                        -- 0 -> 3073 (0b110000000001)
    os                 varchar,
    os_type            varchar(100),                                                                -- windows / linux
    os_version         varchar(100),
    os_bit             integer,
    requester_id       varchar(255),                                                                -- requester ip+uuid
    group_name         varchar(255),                                                                -- unique id (not display name)
    capacity           integer default 1,                                                           -- capacity of current machine
    assign_time        timestamp default CURRENT_TIMESTAMP,                                         -- request timestamp
    release_time       timestamp default CURRENT_TIMESTAMP,                                         -- release timestamp
    timeout_time       timestamp default CURRENT_TIMESTAMP,                                         -- timeout timestamp
    heartbeat_time     timestamp default CURRENT_TIMESTAMP,
    created_time       timestamp default CURRENT_TIMESTAMP,
    lastmodifiedtime   timestamp default CURRENT_TIMESTAMP,
    machine_info       varchar   default '{}'::character varying
);

CREATE OR REPLACE FUNCTION mct.trigger_fct_tr_mct_resource_machines_update()
 RETURNS trigger
 LANGUAGE plpgsql
 SECURITY DEFINER
AS $function$
BEGIN
    NEW.lastmodifiedtime := sysdate();
    RETURN NEW;
END;
$function$
;
CREATE TRIGGER tr_mct_resource_machines_update
BEFORE INSERT OR UPDATE ON mct_resource_machines
FOR EACH ROW
EXECUTE FUNCTION trigger_fct_tr_mct_resource_machines_update();

Machine Heartbeat ¶

Register / update machine

Path: /management/machine/group/{groupName}/register
Method: POST

Body Example:

{
    "ip": "10.224.251.200",
    "hostname": "mctuser02.webex.com",
    "username": "Administrator",
    "testApp": "outlook",
    "port": 4720,
    "capacity": 1,
    "osBit": "64"
}

- Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

groupName can be kubed-cluster name or pool name

Get Machine List ¶

Get machine list, if groupName is not provided, will return all machines

Path: /management/machine?group={groupName}&osType={osType}
Method: GET

Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "192.168.3.8",
    "errorMessage": null,
    "result": [
      {
          "status": 3073,
          "requesterInfos": [
              {
                  "id": 888,
                  "resourceId": 123,
                  "resourceType": "MACHINE",
                  "requesterId": "111",
                  "assignTime": "2025-04-25T03:12:15.339178",
                  "timeoutTime": "2025-04-25T03:12:15.339178"
              }
          ],
          "releaseTime": "2025-02-12T04:44:31",
          "id": 1000000521,
          "hostname": "test.webex.com",
          "ip": "127.0.0.11",
          "username": "test1",
          "port": 4720,
          "groupName": "bt4",
          "lastModifiedTime": "2025-02-12T04:46:15",
          "heartbeatTime": "2025-02-12T12:45:07.991348",
          "os": "windows1123",
          "osType": "WINDOWS",
          "osVersion": "20161123",
          "machineInfo": "{\"chrome\":\"132.0.6834.84\",\"webexapp\":\"45.3.1.32057\"}",
          "osBit": 64,
          "capacity": 1,
          "enable": true,
          "available": true,
          "idle": true,
          "online": true,
          "idleCount": 1
      },
      {
          "status": 3076,
          "requesterInfos": [
              {
                  "id": 777,
                  "resourceId": 122,
                  "resourceType": "MACHINE",
                  "requesterId": "123123123",
                  "assignTime": "2025-04-25T03:12:15.339178",
                  "timeoutTime": "2025-04-25T03:12:15.339178"
              }
          ],
          "releaseTime": "2025-02-12T04:44:31",
          "id": 1000000263,
          "hostname": "testtest.webex.com",
          "ip": "127.0.0.1",
          "username": "test1",
          "port": 4720,
          "groupName": "bt4",
          "lastModifiedTime": "2025-02-12T04:45:29",
          "heartbeatTime": "2025-02-12T12:45:10.444673",
          "os": "windows",
          "osType": "windows",
          "osVersion": "2016",
          "machineInfo": "{\"chrome\":\"132.0.6834.84\",\"webexapp\":\"45.3.1.32057\"}",
          "osBit": 64,
          "capacity": 4,
          "enable": true,
          "available": true,
          "idle": true,
          "online": true,
          "idleCount": 4
      }
    ]
}

groupName can be kubed-cluster name or pool name osType LINUX / WINDOWS

Delete Machine By ID ¶

Delete account by id

Path: /management/machine/{id}
Method: DELETE

Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

Resource Management ¶

Request and release resource (windows machine and account)

Occupation Table DDL ¶

-- PRI
create sequence s_mct_resource_occupation
    minvalue 1000000001
    increment by 2
    maxvalue 9999999999
    cache 10
    cycle;
-- GSB
create sequence s_mct_resource_occupation
    minvalue 1000000000
    increment by 2
    maxvalue 9999999999
    cache 10
    cycle;

create table mct_resource_occupation
(
    id                 bigint default nextval('s_mct_resource_occupation'::regclass) not null primary key, -- the id
    resource_id        bigint not null,                                                             -- resource id
    resource_type      varchar,                                                                     -- machine / account
    requester_id       varchar,                                                                     -- requester ip+uuid
    group_name         varchar   default '',
    assign_time        timestamp default CURRENT_TIMESTAMP,                                         -- request timestamp
    release_time       timestamp default CURRENT_TIMESTAMP,                                         -- release timestamp ? is it needed?
    timeout_time       timestamp default CURRENT_TIMESTAMP,                                         -- timeout timestamp
    lastmodifiedtime   timestamp default CURRENT_TIMESTAMP
);

CREATE OR REPLACE FUNCTION mct.trigger_fct_tr_mct_resource_occupation_update()
    RETURNS trigger
    LANGUAGE plpgsql
    SECURITY DEFINER
AS $function$
BEGIN
    NEW.lastmodifiedtime := sysdate();
    RETURN NEW;
END;
$function$
;
CREATE TRIGGER tr_mct_resource_occupation_update
    BEFORE INSERT OR UPDATE ON mct_resource_occupation
    FOR EACH ROW
EXECUTE FUNCTION trigger_fct_tr_mct_resource_occupation_update();

Request Resource ¶

Request resource (for certain amount), designate a strategy is possible.

Path: /management/resource/v1/requester/{requesterId}/request?strategy={strategy}
Method: POST

Body Example:

{
    "windowsCount": 1,
    "linuxCount": 1,
    "sharedAccountCount": 1,
    "exclusiveAccountCount": 1,
    "appUrl": "https://dummy.webex.com",
    "group": "bt4", // recommended when `windowsCount > 0`
    "testApp": "meeting" // optional
}

- Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "172.16.3.18",
    "result": {
        "winMachines": [
          {
              "status": 3073,
              "requesterInfos": [
                  {
                      "id": 888,
                      "resourceId": 123,
                      "resourceType": "MACHINE",
                      "requesterId": "111",
                      "assignTime": "2025-04-25T03:12:15.339178",
                      "timeoutTime": "2025-04-25T03:12:15.339178"
                  }
              ],
              "releaseTime": "2025-02-12T04:44:31",
              "id": 1000000521,
              "hostname": "test.webex.com",
              "ip": "127.0.0.11",
              "username": "test1",
              "port": 4720,
              "groupName": "bt4",
              "lastModifiedTime": "2025-02-12T04:46:15",
              "heartbeatTime": "2025-02-12T12:45:07.991348",
              "os": "windows1123",
              "osType": "WINDOWS",
              "osVersion": "20161123",
              "machineInfo": "{\"chrome\":\"132.0.6834.84\",\"webexapp\":\"45.3.1.32057\"}",
              "osBit": 64,
              "capacity": 1,
              "enable": true,
              "available": true,
              "idle": true,
              "online": true,
              "idleCount": 1
          },
          {
              "status": 3076,
              "requesterInfos": [
                  {
                      "id": 777,
                      "resourceId": 111,
                      "resourceType": "MACHINE",
                      "requesterId": "111",
                      "assignTime": "2025-04-25T03:12:15.339178",
                      "timeoutTime": "2025-04-25T03:12:15.339178"
                  }
              ],
              "releaseTime": "2025-02-12T04:44:31",
              "id": 1000000522,
              "hostname": "test1.webex.com",
              "ip": "127.0.0.12",
              "username": "test",
              "port": 4720,
              "groupName": "bt4",
              "lastModifiedTime": "2025-02-12T04:46:15",
              "heartbeatTime": "2025-02-12T12:45:07.991348",
              "os": "linux",
              "osType": "LINUX",
              "osVersion": "20161123",
              "machineInfo": "{\"chrome\":\"132.0.6834.84\",\"webexapp\":\"45.3.1.32057\"}",
              "osBit": 64,
              "capacity": 4,
              "enable": true,
              "available": true,
              "idle": true,
              "online": true,
              "idleCount": 4
          }
        ],
        "sharedAccounts": [
          {
              "status": 3,
              "requesterInfos": [],
              "assignTime": "2025-02-12T02:54:51",
              "releaseTime": "2025-02-12T02:57:56",
              "timeoutTime": "2025-02-12T10:57:51.720598",
              "id": 1000000361,
              "testApp": "meeting",
              "appUrls": [
                  "https://dummy.webex.com"
              ],
              "appName": "dummy",
              "username": "zhochi1",
              "password": "xxx",
              "boundMachineIp": null,
              "createdTime": "2025-01-26T16:47:59.847571",
              "lastModifiedTime": "2025-02-12T02:57:56",
              "ready": true,
              "enable": true
          }
        ],
        "exclusiveAccounts": [,
          {
              "status": 2,
              "requesterId": : [
              {
                  "id": 999,
                  "resourceId": 333,
                  "resourceType": "ACCOUNT",
                  "requesterId": "111",
                  "assignTime": "2025-04-25T03:12:15.339178",
                  "timeoutTime": "2025-04-25T03:12:15.339178"
              }
              ],
              "assignTime": "2025-02-11T08:52:29",
              "releaseTime": "2025-02-05T18:00:36.353198",
              "timeoutTime": "2025-02-11T16:55:29.353746",
              "id": 1000000381,
              "testApp": "meeting",
              "appUrls": [
                  "https://dummy.webex.com"
              ],
              "appName": "dummy",
              "username": "zhochi",
              "password": "xxx",
              "boundMachineIp": null,
              "createdTime": "2025-02-05T15:56:30.826293",
              "lastModifiedTime": "2025-02-11T08:52:29",
              "ready": false,
              "enable": true
          }
        ]
    }
}

requesterId can be a random uuid strategy can be one of [LRU, RANDOM]. if not provided, will use LRU as default

Accounts without `boundMachineIp` ¶

Acquire accounts and machines separately

Accounts with `boundMachineIp` ¶

Acquire account, with bound machine is available
Acquire machine by boundMachineIp

Note: the relationship of account and boundMachineIp should be 1:1

Renew Resource Lease ¶

Renew test account (for certain amount), if group is not provided, will get an available machine from a random group

Path: /management/resource/requester/{requesterId}/renew
Method: POST

Body Example:

{
    "duration": 2, // in mins, default is 3
    "resources": [ // resources to be renewed, default linMachines + winMachines + exclusiveAccounts + sharedAccounts
        "winMachines",
        "exclusiveAccounts"
    ]
}

- Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "172.16.3.18"
}

requesterId can be a random uuid this api should be called after resource is acquired, otherwise will got exception

Release Resource ¶

By API ¶

Release resource (windows machine and account) by requesterId

Path: /management/resource/requester/{requesterId}/release
Method: POST

Body Example:

{
    "resources": [ // resources to be released, default linMachines + winMachines + exclusiveAccounts + sharedAccounts
        "winMachines",
        "exclusiveAccounts"
    ]
}

- Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

requesterId can be a random uuid

By Task ¶

Resource should be auto released when requester is inactive for certain time (3 mins). There will be a task to monitor the resource status.

Machine Capacity Management ¶

For Linux cases, a single machine can run multiple instances simultaneously. Each machine is required to report a capacity through heartbeat signal, representing the maximum number of instances it can support. This capacity information is stored in the database as a 10-bit binary value (1024).

When allocating machine resources, we try to select those with a positive capacity (idle state). If a machine's capacity is greater than 1, the capacity is decremented by 1 upon allocation, and the instance is returned. When the capacity value is reduced to 0, the machine is considered to be in a busy state.

State Transition Diagram ¶

Prometheus Exporter ¶

Add a prometheus exporter, to get get current resource status.

Account: 1. total account count 2. enable account count 3. disable account count 4. ready account count 5. busy account count

Machine: 1. total machine count 2. enable machine count 3. disable machine count 4. ready machine count 5. busy machine count 6. online machine count 7. offline machine count

Test Case Management ¶

Add / Update Test Case ¶

Path: /management/test-case
Method: POST

Body Example:

{
  "caseVersion": "0.1.1.2",
  "caseName": "test12",
  "caseYamlBody": "abca1111ssss22222"
}

- Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

Get All Test Cases ¶

Get test case list

Path: /management/test-case
Method: GET

Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "172.16.3.18",
    "result": [
        {
            "id": 1,
            "caseVersion": "0.1",
            "caseName": "test12",
            "caseYamlBody": "abca1111ssss2",
            "createdTime": 1729910073085,
            "lastModifiedTime": null
        },
        {
            "id": 3,
            "caseVersion": "0.1.1",
            "caseName": "test12",
            "caseYamlBody": "abca1111ssss2",
            "createdTime": 1729910078302,
            "lastModifiedTime": null
        }
    ]
}

Get Test Cases By name ¶

Get test case list by name

Path: /management/test-case/test12
Method: GET

Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "172.16.3.18",
    "result": [
        {
            "id": 1,
            "caseVersion": "0.1",
            "caseName": "test12",
            "caseYamlBody": "abca1111ssss2",
            "createdTime": 1729910073085,
            "lastModifiedTime": null
        },
        {
            "id": 3,
            "caseVersion": "0.1.1",
            "caseName": "test12",
            "caseYamlBody": "abca1111ssss2",
            "createdTime": 1729910078302,
            "lastModifiedTime": null
        }
    ]
}

Delete Test Cases By id ¶

Delete test case by id

Path: /management/test-case/{id}
Method: DELETE

Response Example:

{
    "errorCode": "OKOKOK",
    "localIp": "172.16.3.18"
}

Reference ¶

Below are the Refactor Discussion Output

Centralize or Distributed ¶

Centralize advantage:

share machine, increase usage rate
Better for future maintenance

difference:

add namespace for machine
win machine communicate with resource manager directly, add a location config (kubed-cluster name or pool name)
plugin add a location config (kubed-cluster name or pool name)

Machine ¶

add a location config (kubed-cluster name or pool name)

Account ¶

account bind to meeting site, instead of site url. maintain site-urls mapping on DB

Resource apply logic ¶

add a resource declare logic at the beginning, parameters can be a placeholder, such as site url. (control hub case is not included)
plugin generate a uuid
request a machine with uuid, and resource manager bind the machine with this uuid
request account, and resource manager bind the account with this uuid

Resource Manager Refactor

Key Points ¶

Overall architecture ¶

Account Management ¶

Account Table DDL ¶

Create Account ¶

Update Account ¶

Get Account List ¶

Delete Account By ID ¶

Windows Machine Management ¶

Machine Table DDL ¶

Machine Heartbeat ¶

Get Machine List ¶

Delete Machine By ID ¶

Resource Management ¶

Occupation Table DDL ¶

Request Resource ¶

Accounts without boundMachineIp ¶

Accounts with boundMachineIp ¶

Renew Resource Lease ¶

Release Resource ¶

By API ¶

By Task ¶

Machine Capacity Management ¶

State Transition Diagram ¶

Prometheus Exporter ¶

Test Case Management ¶

Add / Update Test Case ¶

Get All Test Cases ¶

Get Test Cases By name ¶

Delete Test Cases By id ¶

Reference ¶

Centralize or Distributed ¶

Machine ¶

Account ¶

Resource apply logic ¶

Accounts without `boundMachineIp` ¶

Accounts with `boundMachineIp` ¶