Kube Job Management PoC and Design

Background ¶

This document outlines the design for converting an image executor running on a host machine into Kubernetes Job. Job is one-time workload. It creates one or more pods which run to completion with wrapped specific images you want to execute. This transition leverages the scalability, reliability, and orchestration capabilities of Kubernetes to manage workloads efficiently.

Architecture Overview ¶

Current Solution ¶

Image Executor on Host Machine ¶

Pull images from pointed registry
Run container images directly using Docker with command line and parameters (if any)
Catch container exit code and log output

Target Solution ¶

Kubernetes Job ¶

Job manager controls Job handling, e.g., creation, deletion, list and etc
Each Job encaplates a image, input parameters (if any) and ConfigMap (if any)

Components ¶

This section consists of all necessary components, not only applictions, but resources in Kubernates, for completing the Target Solution. First of all, we introuduce a work diagram to show the brief work logic. And then, we descsirbe more details of each components.

Work Diagram ¶

sequenceDiagram
    participant Plugin
    participant Job Manager
    participant Kube API

    Plugin->>Job Manager: Image and Command Line
    Job Manager->>Kube API: Create Job
    Kube API->>Job Manager: Job Creation/Running Response
    Job Manager->>Job Manager: manage Job metadata locally
    Job Manager->>Kube API: Query Job Execution Result
    Kube API->>Job Manager: Response Job Execution Result
    Job Manager->>Plugin: Job Execution Result

Service Account ¶

Here we employes Service Account in two purposes: 1. provide an identity to authenticate to Kubernates API server. Here is example of creating a service account:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-job-manager-service-account
  namespace: mas-monitor-worker

2. provide a way to grant image pull permission to service account. Here is example of creating image pull secret and appending such permission to ServiceAccount:

docker_registry_config=$(echo -n '{"auths":{"<registry name>":{"username":"DUMMY_USERNAME","password":"DUMMY_DOCKER_PASSWORD","email":"DUMMY_DOCKER_EMAIL"}}}' | base64 -w0)

apiVersion: v1
kind: Secret
metadata:
  name: kube-job-manager-image-registry-<registry name>-secret
  namespace: mas-monitor-worker
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <docker_registry_config>
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-job-manager-service-account
  namespace: mas-monitor-worker
imagePullSecrets:
- name: kube-job-manager-image-registry-<registry name>-secret

Role ¶

RBAC Role represents a set of permissions within a particular namespace. Here is example of granting creation and read access to Pods:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: mas-monitor-worker
  name: kube-job-manager-role
rules:
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "create", "watch"]

RoleBinding ¶

RBAC RoleBinding grants the permissions defined in a Role to a user or set of groups. It holds a list of subjects (users, groups, or service accounts) and a reference to a Role. Here is example of binding the service account to the role created previously:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kube-job-manager-rolebinding
  namespace: mas-monitor-worker
subjects:
- kind: ServiceAccount
  name: kube-job-manager-service-account
  namespace: mas-monitor-worker
roleRef:
  kind: Role
  name: kube-job-manager-role
  apiGroup: rbac.authorization.k8s.io

Kube Job Manager ¶

Kube Job Manager (PoC) is running as a service, assigned permission based on the Service Account setting, communicating with Kube API server with and controlling Job operations, e.g., creating and deleting Job, according to the input requests. The application encapsulates image, entrypoint (if any), ConfigMap (if any) and Secret(if any) as Job spec based on predefined Job template, and then communicate with Kube API to manage Jobs within the same namespace.

Plugin of MCT ¶

In order to better integrate with MCT architecture, a plugin is developed as adaptor to receive container setting and to communicate with Kube Job Manager controlling Job lifecycle.

ConfigMap and Secrets (If Any) ¶

Sometimes, you expect to pass configuration file and secret informations into container. In this case, you can create a Configmap and Secret mounted to the image executor as files. Here is example of creating ConfigMap and Secret, and binding to a Job:

# ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: dc-01awxt1-tap-<team>-config-example
  namespace: mas-monitor-worker
data:
  # file-like keys
  sites: |
    dfw=test-dfw.webex.com
    jfk=test-jfk.webex.com
---
# Secret
apiVersion: v1
kind: Secret
metadata:
  name: dc-01awxt1-tap-<team>-secret-example
  namespace: mas-monitor-worker
data:
  # property-like keys; each key maps to a simple value
  username: "test-user"
  password: "***"
---
# Job
apiVersion: batch/v1
kind: Job
metadata:
  name: tap-job-<team>-<uuid>-example
  namespace: mas-monitor-worker
spec:
  template:
    spec:
      containers:
      - name: list-files
        image: bash:lastest
        command: ["/bin/sh", "-c", "ls -la /etc/config; ls -al /etc/secret"]
        volumeMounts:
        - name: tap-job-<team>-<uuid>-config-example
          mountPath: /etc/config
          # Configmape as file read-only mode
          readOnly: true
        - name: tap-job-<team>-<uuid>-secret-example
          mountPath: /etc/secret
      restartPolicy: Never
      volumes:
      - name: tap-job-<team>-<uuid>-config-example
        configMap:
          name: dc-01awxt1-tap-<team>-config-example
      - name: tap-job-<team>-<uuid>-secret-example
        secret:
          secretName: dc-01awxt1-tap-<team>-secret-example