Skip to content

Common Check Provision API Design

Background

Check is the base unit of the MCT platform, but registering a check is not straightforward:

  1. Users must learn certain knowledge about MCT, such as cluster, zone, check type, timer, etc.
  2. Users need to create or find the entire DC / Service -> Cluster -> Zone -> Check structure to add a single check.

Since check provision is a critical module of MCT, we need to provide a convenient approach to register / decommission checks.

Mind Map

Currently, we have PA/PV check registration and kubed app registration, which are based on two parts:

  1. An infra file, mainly a list of applications or servers, with some static properties like kubed_cluster, env, dc, etc.
  2. A template YAML file to map the above applications/servers to MCT units (cluster, zone, check), which may have some placeholders with static properties.

A GitHub webhook, an MQ message, or an API call can trigger the registration process, combining the above two parts into a complete check structure.

This is a good practice for both end users and MCT engineers. Now, we need to extend it to a more common approach.

Solution Detail

Template File

The template is a yaml format file. We leverage Pebble as the template engine, which is similar to Django. It is easy to get started, especially for users with basic CMC template knowledge.

Template definition should be done prior to check registration / decommission, and it is a one-time task. Each kind of server/application needs to have a separate template to map them to MCT checks. This step usually requires cooperation between MCT engineers and MCT customers, which is a one-time job.

The template is now persistent in the MCT DB and can be migrated to a GitHub template repository in a later phase.

Below is a brief example of a template (tap-setting) with Pebble placeholders enabled:

# {#
# Meta:
#   creator: zhochi@cisco.com
#   serviceName: MAS-Monitoring
#   backstageLink: https://backstage.corp.webex.com/catalog/default/system/kubed-monitoring-stack-system
# 
# Parameter definitions:
#   1. appGroup: the owner group of the tap case. e.g. calendar
#   2. env: the environment. e.g. prod
#   3. kubedCluster: the cluster where the current tap case is located. e.g. wfraint
#   4. caseName: the tap case name. e.g. cloud-connect-test
#   5. imageName: the tap case image name. e.g. containers.cisco.com/webexsquared/calendar-cloud-connector-test:7758-240715-d26cf
#   6. commandLine: entry point of the tap case. e.g. /usr/local/run.sh
#   7. environmentVariables: environment variables when running the tap case. e.g. isE2eOnly=true
#   8. configMountPath: path of configMap resource. e.g. /opt/config
#   9. secretMountPath: path of secret resource. e.g. /opt/config
#   10. resultFilename: location of the tap case's output. e.g. /test-output/junitreports/TEST-com.ciscospark.calendarconnector.integration.IntegrationTest.xml
#   11. timeout: execution timeout, in minutes. e.g. 5
# #}
global_setting:
    datacenter: "TAP"
    cluster_name: "TaP-{{Values.appGroup}}"
    service: 10000007 #Tools
    env: Prod
server_type_list:
    -  zone_setting:
            domainType: 101 #Plugin
            zoneType: "KubernetesJobExecutor" #KubernetesJobExecutor
            zoneTypeName: "KubernetesJobExecutor"
            zoneName: "{{Values.env}}-{{Values.kubedCluster}}"
            timerInterval: "5"      #min
        server_setting:
            serverType: "KubernetesJobExecutor"
            svrType: "tap-settings"
            serverIp: "{{Values.caseName}}"
            serverHostname: "{{Values.caseName}}"
            image: "{{Values.imageName}}"
            command_line: "{{Values.commandLine | join('\n')}}"
            environment_variables: "{{Values.environmentVariables | join('\n')}}"
            configmap_mount_path: "{{Values.configMountPath}}"
            secret_mount_path: "{{Values.secretMountPath}}"
            result_filename: "{{Values.resultFilename}}"
            timeout: "{{Values.timeout}}"
            method: 0 #Get

Some highlights: 1. The template should have a unique name (usually the application name, here it is tap-setting). Each tap-setting will be mapped to an MCT check with type KubernetesJobExecutor. 2. All MCT-related logic is reflected in YAML keys, such as data center, service, branch, zone type, and other parameters related to separate check types. 3. Pebble template grammar is used; users need to pass those placeholder values to complete the structure render. In other words, users only need to care about template placeholder values. 4. Certain filters and functions can be applied to those placeholder values. 5. It is recommended to add value descriptions at the top of the template, which can serve as a user manual.

Infra List

Infra is a list of applications (or servers). Only those applications with predefined templates can proceed with the registration process. The application name should align with the template name, and each application should contain all placeholder values defined in the template.

Below is an example infra payload regarding the application tap-setting:

{
  "timerInterval": 5, // test frequency
  "appGroup": "calendar", // the owner group of the app
  "validateResult": false, // if validate the check result after registration (manual test)
  "labels": { // labels bind to the current case, which is useful for notification rule setting
    "foo": "bar"
  },
  "parameters": { // customized parameters
    "env": "prod",
    "kubedCluster": "wfraint",
    "caseName": "cloud-connect-test",
    "imageName": "containers.cisco.com/webexsquared/calendar-cloud-connector-test:7758-240715-d26cf",
    "commandLine": "/usr/local/run.sh",
    "environmentVariables": "isE2eOnly=true",
    "configMountPath": "/opt/config",
    "secretMountPath": "/opt/config",
    "resultFilename": "/test-output/junitreports/TEST-com.ciscospark.calendarconnector.integration.IntegrationTest.xml"
  }
}

Apart from timerInterval, env, and labels, each application may have a different parameter map, depending on the values used in the template.

API Definition

Based on the above two parts, we provide a common API for check registration / decommission.

POST /check/registration/${appName}

Description

Check registration with the whole structure. The appName should be the same as the template name.

Body
{
  "timerInterval": 5, // test frequency
  "appGroup": "calendar", // the owner group of the app
  "validateResult": false, // if validate the check result after registration (manual test)
  "labels": { // labels bind to the current case, which is useful for notification rule setting
    "foo": "bar"
  },
  "parameters": { // customized parameters
    "env": "prod",
    "kubedCluster": "wfraint",
    "caseName": "cloud-connect-test",
    "imageName": "containers.cisco.com/webexsquared/calendar-cloud-connector-test:7758-240715-d26cf",
    "commandLine": "/usr/local/run.sh",
    "environmentVariables": "isE2eOnly=true",
    "configMountPath": "/opt/config",
    "secretMountPath": "/opt/config",
    "resultFilename": "/test-output/junitreports/TEST-com.ciscospark.calendarconnector.integration.IntegrationTest.xml"
  }
}
Response

  • 200 OK

    {
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

  • 400 Bad Request

    {
    "errorCode": "INVALID_INPUT_PARA",
    "localIp": "10.140.212.59",
    "errorMessage": "Failed to process auto register, Template attribute(s) not satisfied, please provide all required parameters: [env]",
}

  • 500 Server Internal Error

    {
    "errorCode": "NONONO",
    "localIp": "10.140.212.59",
    "errorMessage": "Internal error"
}

POST /check/decommission/${appName}

Description

Check decommission with the whole structure. The appName should be the same as the template name.

Body
{
  "timerInterval": 5, // test frequency
  "appGroup": "calendar", // the owner group of the app
  "validateResult": false, // if validate the check result after registration (manual test)
  "labels": { // labels bind to the current case, which is useful for notification rule setting
    "foo": "bar"
  },
  "parameters": { // customized parameters
    "env": "prod",
    "kubedCluster": "wfraint",
    "caseName": "cloud-connect-test",
    "imageName": "containers.cisco.com/webexsquared/calendar-cloud-connector-test:7758-240715-d26cf",
    "commandLine": "/usr/local/run.sh",
    "environmentVariables": "isE2eOnly=true",
    "configMountPath": "/opt/config",
    "secretMountPath": "/opt/config",
    "resultFilename": "/test-output/junitreports/TEST-com.ciscospark.calendarconnector.integration.IntegrationTest.xml"
  }
}
Response

  • 200 OK

    {
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59"
}

  • 400 Bad Request

    {
    "errorCode": "INVALID_INPUT_PARA",
    "localIp": "10.140.212.59",
    "errorMessage": "Failed to process auto decommission, Template attribute(s) not satisfied, please provide all required parameters: [env]",
}

  • 500 Server Internal Error

    {
    "errorCode": "NONONO",
    "localIp": "10.140.212.59",
    "errorMessage": "Internal error"
}

POST /check/registration

Description

Check registration with the whole structure async, it requires a mail or a teams room id to receive register result.

Body
{
  "contact": "zhochi@cisco.com", // user mail address, or a teams roomId, used to receive notification
  "globalConfig": { // global configs
    "timerInterval": 1, // test frequency
    "appGroup": "calendar", // the owner group of the app
    "validateResult": false, // if validate the check result after registration (manual test)
    "labels": { // labels bind to the current case, which is useful for notification rule setting
      "foo": "bar"
    },
    "parameters": { // customized parameters
      "env": "prod",
      "kubedCluster": "wfraint",
    }
  },
  "appConfigs": [ // application list with separate config
    {
      "appName": "tap-setting",
      "timerInterval": 5, // test frequency
      "labels": { // labels bind to the current case, which is useful for notification rule setting
        "fooInner": "barInner"
      },
      "parameters": { // customized parameters
        "env": "prod",
        "kubedCluster": "wfraint",
        "caseName": "cloud-connect-test",
        "imageName": "containers.cisco.com/webexsquared/calendar-cloud-connector-test:7758-240715-d26cf",
        "commandLine": "/usr/local/run.sh",
        "environmentVariables": "isE2eOnly=true",
        "configMountPath": "/opt/config",
        "secretMountPath": "/opt/config",
        "resultFilename": "/test-output/junitreports/TEST-com.ciscospark.calendarconnector.integration.IntegrationTest.xml"
      }
    }
  ]
}

Global common values can be put on globalConfig field, values in appConfigs will override global level value.

Response

  • 200 OK

    {
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59",
    "result": "Register task submitted, please pay attention to Teams notifications"
}

  • 400 Bad Request

    {
    "errorCode": "INVALID_INPUT_PARA",
    "localIp": "10.140.212.59",
    "errorMessage": "Error validating payload: appName is mandatory."
}

  • 500 Server Internal Error

    {
    "errorCode": "NONONO",
    "localIp": "10.140.212.59",
    "errorMessage": "Internal error"
}

POST /check/decommission

Description

Check decommissio with the whole structure async, it requires a mail or a teams room id to receive decommission result.

Body
{
  "contact": "zhochi@cisco.com", // user mail address, or a teams roomId, used to receive notification
  "globalConfig": { // global configs
    "timerInterval": 1, // test frequency
    "appGroup": "calendar", // the owner group of the app
    "validateResult": false, // if validate the check result after registration (manual test)
    "labels": { // labels bind to the current case, which is useful for notification rule setting
      "foo": "bar"
    },
    "parameters": { // customized parameters
      "env": "prod",
      "kubedCluster": "wfraint",
    }
  },
  "appConfigs": [ // application list with separate config
    {
      "appName": "tap-setting",
      "timerInterval": 5, // test frequency
      "labels": { // labels bind to the current case, which is useful for notification rule setting
        "fooInner": "barInner"
      },
      "parameters": { // customized parameters
        "env": "prod",
        "kubedCluster": "wfraint",
        "caseName": "cloud-connect-test",
        "imageName": "containers.cisco.com/webexsquared/calendar-cloud-connector-test:7758-240715-d26cf",
        "commandLine": "/usr/local/run.sh",
        "environmentVariables": "isE2eOnly=true",
        "configMountPath": "/opt/config",
        "secretMountPath": "/opt/config",
        "resultFilename": "/test-output/junitreports/TEST-com.ciscospark.calendarconnector.integration.IntegrationTest.xml"
      }
    }
  ]
}

Global common values can be put on globalConfig field, values in appConfigs will override global level value.

Response

  • 200 OK

    {
    "errorCode": "OKOKOK",
    "localIp": "10.140.212.59",
    "result": "Decommission task submitted, please pay attention to Teams notifications"
}

  • 400 Bad Request

    {
    "errorCode": "INVALID_INPUT_PARA",
    "localIp": "10.140.212.59",
    "errorMessage": "Error validating payload: appName is mandatory."
}

  • 500 Server Internal Error

    {
    "errorCode": "NONONO",
    "localIp": "10.140.212.59",
    "errorMessage": "Internal error"
}

GET /check/template/${appName}

Description

Get the template of appName.

Response

  • 200 OK

    {
    "errorCode": "OKOKOK",
    "localIp": "192.168.1.10",
    "errorMessage": null,
    "result": {
        "description": "# {#\n# Meta:\n#   creator: zhochi@cisco.com\n#   serviceName: MAS-Monitoring\n#   backstageLink: https://backstage.corp.webex.com/catalog/default/system/kubed-monitoring-stack-system\n# \n# Parameter definitions:\n#   1. env: the environment. e.g. prod\n#   2. hostname: the hostname of the application. e.g. mct.webex.com\n#   3. appName: application name. in current case, it is `central-api`\n# #}\n",
        "templateYaml": "global_setting:\n  datacenter: \"HF1\"\n  cluster_name: \"MAS_POC\"\n  service: \"Monitoring\"\n  env: \"{{ Values.env }}\"\n\nserver_type_list:\n  - zone_setting:\n      domainType: 8 #URL Watch\n      zoneType: \"URL Watch Zone\" #URL Watch Zone\n      zoneName: \"MAS-Monitor-API\"\n      zoneField3: 0\n      zoneField6: 10 #Muti-thread Num\n    server_setting:\n      serverType: \"URL Watch Server\"\n      svrType: \"mas-monitoring\"\n      serverName: \"MAS-Central-API\"\n      url: \"https://{{ Values.hostname }}/mpi/healthcheck\"\n      positive: \"OKOKOK\"\n      negative: \"\"\n      timeout: 60\n      retry: 1\n      mode: 0\n      method: 0 #Get\n  - zone_setting:\n      domainType: 101\n      zoneType: \"ESCountMonitor\"\n      zoneName: \"MAS-ES-Log\"\n      autoMode: 2\n    server_setting:\n      serverType: \"ESCountMonitor\"\n      svrType: \"service\"\n      serverIp: \"{{ Values.appName | upper }} - Central Api Access Log\"\n      serverHostname: \"{{ Values.appName | upper }} - Central Api Access Log\"\n      baseUrl: \"https://api-logs.o.webex.com/\"\n      topic: \"wbx2-access:logs*\"\n      syntax: 1\n      dsl: \"{\\\"query\\\":{\\\"match_all\\\":{\\\"datacenter\\\":\\\"{{Values.appName}}\\\"}}}\"\n      times: 1\n      export_metrics: 1\n",
        "attributes": [
            "hostname",
            "appName",
            "env"
        ]
    }
}

  • 400 Bad Request

    {
    "errorCode": "INVALID_INPUT_PARA",
    "localIp": "10.140.212.59",
    "errorMessage": "Cannot find template with name 'tap-set'. Please check your parameters or contact MCT engineers to define it first."
}

  • 500 Server Internal Error

    {
    "errorCode": "NONONO",
    "localIp": "10.140.212.59",
    "errorMessage": "Internal error"
}

Conclusion

Follow these steps to onboard an application on MCT: 1. Define a monitor list for the application, such as HTTP Check and TCP Check. - MCT User, one-time job 2. Provide a parameter list that may be used for monitoring purposes, such as Hostname, Service Port, Endpoint Address. - MCT User, one-time job 3. Define a template based on the above information. - MCT Engineer, one-time job 4. Call the MCT API with the information in step #2, and MCT will provision a check list for the provided application. - MCT User