MAV Service API Migration

Background ¶

MAV (Monitoring Agent Version) service is being retired. Its APIs are consumed by monitoring agents (telegraf / tel-conf) to obtain Prometheus remote-write endpoints, credentials, and metric filter rules. These APIs need to be migrated to MCT central-api so that clients can switch to the MCT endpoint with minimal disruption.

Migration Principles ¶

The migration follows several key principles to ensure a seamless, user-transparent transition:

• Behavioral parity: MCT API responses must be byte-for-byte identical to the original MAV API responses. Every field — URL, credentials, namepass list, fieldpass list, error messages — must match exactly.
• URL compatibility: The original MAV API path /mavs/pop/... must continue to work on MCT. A Tomcat Valve-based URL rewrite (MavUrlRewriteConfiguration.java) transparently rewrites /mavs/* to /mpi/mavs/* at the engine level, so clients do not need to know about MCT's /mpi context path.
• No authentication change: MAV APIs are unauthenticated. MCT's CI OAuth is disabled at the controller level (@CIOauth(enable = false)) so the migrated APIs remain open, matching the original behavior.
• Data integrity: All MAV MySQL data is migrated to MCT PostgreSQL with INSERT ... ON CONFLICT DO NOTHING to handle MySQL's case-insensitive collation duplicates safely.
• Case-insensitive query behavior: MySQL uses utf8_general_ci (case-insensitive). PostgreSQL is case-sensitive by default. The MCT MyBatis queries use LOWER() on both sides of DC name comparisons to preserve the original case-insensitive lookup behavior.

MAV API Overview ¶

MAV exposes 5 APIs under /mavs/pop.

API 1 & 2 — Core Prometheus Config Lookup ¶

This is the most critical API. It implements a 4-level cascade lookup from the lma_prometheus_detail table:

1. Exact match: env + svrtype + dc + environment
2. Fallback: env + dc + environment (svrtype = 'all')
3. Fallback: svrtype + dc + environment (env = 'all')
4. Fallback: dc + environment (env = 'all', svrtype = 'all')

Meaning:

• Exact match means the query first looks for the most specific row where all four dimensions match the request exactly.
• Fallback means if that exact row does not exist, the query relaxes one condition at a time by using the shared default value all, so the API can still return a valid broker config for the same DC and environment.

Example compare case:

# MAV
curl -sk -X GET \
    'https://mavservice.webex.com/mavs/pop/getlmaprometheus' \
    -H 'Content-Type: application/json' \
    -d '{"host":"test-host","type":"install","dc":"SJC02","svrtype":"linus","env":"PROD","environment":"COMMERCIAL","env_type":"PROD"}' \
    | python3 -m json.tool

# MCT
curl -sk -X GET \
    'https://mctapi.webex.com/mavs/pop/getlmaprometheus' \
    -H 'Content-Type: application/json' \
    -d '{"host":"test-host","type":"install","dc":"SJC02","svrtype":"linus","env":"PROD","environment":"COMMERCIAL","env_type":"PROD"}' \
    | python3 -m json.tool

Note: for MAV/MCT comparison, the tested GET form sends a JSON request body. Calling this endpoint with only query parameters returns a bad request.

POST compare case for the same lookup:

# MAV
curl -sk -X POST \
    'https://mavservice.webex.com/mavs/pop/getlmaprometheus' \
    -H 'Content-Type: application/json' \
    -d '{"host":"test-host","type":"install","dc":"SJC02","svrtype":"linus","env":"PROD","environment":"COMMERCIAL","env_type":"PROD"}' \
    | python3 -m json.tool

# MCT
curl -sk -X POST \
    'https://mctapi.webex.com/mavs/pop/getlmaprometheus' \
    -H 'Content-Type: application/json' \
    -d '{"host":"test-host","type":"install","dc":"SJC02","svrtype":"linus","env":"PROD","environment":"COMMERCIAL","env_type":"PROD"}' \
    | python3 -m json.tool

It also queries lma_namepass and lma_fieldpass for metric filter rules (including common svrtype rules), and asynchronously logs agent version information to agent_status and agent_version_history.

API 3 — Prometheus URL Only ¶

A simplified variant that returns only the url field using the same cascade lookup logic.

Example compare case:

# MAV
curl -sk -X GET \
    'https://mavservice.webex.com/mavs/pop/getlmaprometheusurl' \
    -H 'Content-Type: application/json' \
    -d '{"host":"test-host","type":"install","dc":"SJC02","svrtype":"linus","env":"PROD","environment":"COMMERCIAL","env_type":"PROD"}' \
    | python3 -m json.tool

# MCT
curl -sk -X GET \
    'https://mctapi.webex.com/mavs/pop/getlmaprometheusurl' \
    -H 'Content-Type: application/json' \
    -d '{"host":"test-host","type":"install","dc":"SJC02","svrtype":"linus","env":"PROD","environment":"COMMERCIAL","env_type":"PROD"}' \
    | python3 -m json.tool

API 4 — Query Metrics Filter ¶

Returns the namepass and fieldpass lists for a given svrtype, merged with the common svrtype rules.

Example compare case:

# MAV
curl -sk -X GET \
    'https://mavservice.webex.com/mavs/pop/query-metrics-filter?svrtype=linus' \
    | python3 -m json.tool

# MCT
curl -sk -X GET \
    'https://mctapi.webex.com/mavs/pop/query-metrics-filter?svrtype=linus' \
    | python3 -m json.tool

API 5 — Add Metrics Filter ¶

Inserts new namepass/fieldpass entries for a svrtype. Validates that the svrtype is not null/empty.

requester is also required for successful write requests.

Warning: this API is a write operation. Running the curl examples below will modify MAV or MCT database data by inserting new namepass or fieldpass rows for the specified svrtype.

Use only a disposable test svrtype such as _test_migr_tmp, and clean up the inserted rows after verification.

Example compare case:

# MAV
curl -sk -X POST \
    'https://mavservice.webex.com/mavs/pop/add-metrics-filter' \
    -H 'Content-Type: application/json' \
    -d '{"svrtype":"_test_migr_tmp","requester":"migration-test","namepass":["test_cpu","test_mem"]}' \
    | python3 -m json.tool

# MCT
curl -sk -X POST \
    'https://mctapi.webex.com/mavs/pop/add-metrics-filter' \
    -H 'Content-Type: application/json' \
    -d '{"svrtype":"_test_migr_tmp","requester":"migration-test","namepass":["test_cpu","test_mem"]}' \
    | python3 -m json.tool

Fieldpass compare case:

# MAV
curl -sk -X POST \
    'https://mavservice.webex.com/mavs/pop/add-metrics-filter' \
    -H 'Content-Type: application/json' \
    -d '{"svrtype":"_test_migr_tmp","requester":"migration-test","fieldpass":["test_usage_idle","test_used_percent"]}' \
    | python3 -m json.tool

# MCT
curl -sk -X POST \
    'https://mctapi.webex.com/mavs/pop/add-metrics-filter' \
    -H 'Content-Type: application/json' \
    -d '{"svrtype":"_test_migr_tmp","requester":"migration-test","fieldpass":["test_usage_idle","test_used_percent"]}' \
    | python3 -m json.tool

For API 5, use a disposable svrtype like _test_migr_tmp and clean it up after verification. Do not run these examples directly against prod unless you intend to write and then remove test data.

MCT Migration Implementation ¶

Code Changes ¶

All migration code lives in mct-central-api:

Database Schema ¶

5 MySQL tables are mapped to 5 PostgreSQL tables:

Key Bug Fix — Case Sensitivity ¶

MySQL's utf8_general_ci collation treats DFW02 and dfw02 as the same value. PostgreSQL does not. This caused 4 test cases to fail in early test rounds.

Fix: Added LOWER() to all DC name comparisons in MavDao.xml:

LOWER(dc_name) = LOWER(#{dcName})

This fix is applied to all 4 cascade-level queries, ensuring the same case-insensitive behavior as the original MySQL queries.

URL Rewrite — /mavs/* Without /mpi Prefix ¶

MCT uses server.servlet.context-path=/mpi, but MAV clients call /mavs/pop/... directly. Removing the context path would affect all other MCT APIs.

Solution: MavUrlRewriteConfiguration.java registers a Tomcat ValveBase at the engine level. It intercepts requests with URI starting with /mavs/ and remaps them to /mpi/mavs/... internally, before the servlet container routes the request.

Both paths work after this change:

• https://<mct-host>/mavs/pop/getlmaprometheus → 200
• https://<mct-host>/mpi/mavs/pop/getlmaprometheus → 200

Test Results ¶

Test Methodology ¶

An automated test script calls both MCT and MAV endpoints simultaneously with identical parameters, then compares responses field-by-field:

• JSON fields compared individually: url, usr, pss, orgId
• List fields (namepass, fieldpass) compared as sets (order-insensitive)
• Error messages compared as exact strings
• Write tests (API 5) use a disposable svrtype _test_migr_tmp with automatic cleanup

31 Test Cases ¶

Environment Test Summary ¶

All read-only API comparisons produce identical responses between MCT and MAV across all five environments. Full write-test validation was completed in WFRA and Prod JFK.

WFRA — Full 31/31 ¶

WFRA was the first environment tested. All 31 test cases passed, including 3 write tests (API 5). The write tests insert test data, verify the response matches MAV, then clean up.

BTS JFK — 28/28 ¶

BTS JFK data was manually copied from MAV MySQL using bts-jfk/01_copy_data.py. 28 read-only tests passed. Write tests were skipped because they require a local MySQL connection for cleanup verification.

BTS DFW — 28/28 ¶

BTS DFW receives its data from BTS JFK via SharePlex replication (managed by DBA). No manual data copy was needed. 28 read-only tests passed, confirming the SharePlex-sync'd data is fully consistent.

Prod JFK — 31/31 ¶

Prod JFK data was manually copied from MAV MySQL to PostgreSQL. All 31 test cases passed, including 3 write tests for add-metrics-filter. The write tests used disposable svrtype _test_migr_tmp and cleaned up after each case.

Prod DFW — 28/28 ¶

Prod DFW receives its data from Prod JFK via SharePlex replication. No separate manual data copy was needed. 28 read-only tests passed, confirming the replicated data is consistent with MAV and with Prod JFK.