Release Process Kubed

Overview ¶

This document is about MCT monthly Release process in Kubed.

In short, the process is: build image -> merge change -> argo deploy.

Build application image ¶

Build application docker image with jenkins job and record image tag name in job output.

Application jenkins job ¶

MCT Service Build Pipeline: central-api, status-api, job-engine, status-consumer

MCT Portal Kubed Build Pipeline: web-new-portal

Image tag name ¶

Example in MCT-Kubed-Dev teams room: (20.0.0-4)

Validate pipeline job parameters - Approver (Eden Wang) 
    •   Package service:              job-engine,central-api 
    •   Tag name:                 20.0.0-4

Example in job console log: (20.0.0-4)

    "image_name": "mas/mct-central-api",
    "image_tag": "20.0.0-4",

Merge version change ¶

Following is a case for MCT production configuration initialization change.

kubed-bundles-config ¶

If there are any changes in the kubed-charts repo, you need to update the chart version (keep it the same as defined in kubed-charts)

File: kubed-bundles-config/monitor/values.yaml
    7: api:
    8:   release: mas-central-api
    9:   chartVersion: 1.0.45
    -> 1.0.46

kubed-app-configs ¶

Update image tag name

File: kubed-app-configs/monitor/central-api/values.yaml
  139: image:
  141:   tag: 20.1.1-1
    -> 20.1.1-2

Update commerical application configuration here based on needs.

File: kubed-app-configs/monitor/central-api/prod/values.yaml
   13: # micro-service
   14: microservice_environments: production,govcloud
   -> new add

kubed-app-configs-template ¶

Update PSF application configuration here based on needs.

File: kubed-app-configs-template/templates/monitor/central-api/values.yaml.tpl
   20: kubed:
   21:   cert:
   22:     keyPassword: ''
   23:     location: cert/kubed/eng/client.p12
   -> cert/kubed/prod/client.p12

Application deployment ¶

In short, comments on a empty PR will trigger automatically deployment.

Create pull request for kubed-app-configs repository ¶

Step

Create a brand base latest main branch
Create a empty commit (no code change)
Create pull request to main

Identify target deployment information in branch name and commit message, e.g environment, application name, deploy date, deploy version.

Here is a production central-api git command case:

git checkout main && git pull && \
git checkout -b deploy-mct-prod-central-api-20241101 && \
git commit -m "deploy-mct-prod-central-api-20241101" --allow-empty && \
git push --set-upstream origin deploy-mct-prod-central-api-20241101

Comment with argo command ¶

Notice: comment with "Files changed" -> "Review changes".

Step

Get pull request approved (not merge)
Comment on the pull request with target application argo command
Wait argo automaticlly deplopyment complete

Here is production argo command for each application:

central-api: argo deploy monitor/api.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-gsb
status-api: argo deploy monitor/gpi.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-gsb
job-engine: argo deploy monitor/jobEngine.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-gsb
status-consumer: argo deploy monitor/akka.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-gsb
resource-management: argo deploy monitor/resourceManagement.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-gsb
web-new-portal: argo deploy monitor/webNewPortal.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-gsb
worker-agent: argo deploy monitor/workerAgent.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-dfw02
worker-adapter-java: argo deploy monitor/workerAdapterJava.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-dfw02
appium-runner: argo deploy monitor/appiumRunnerWorker.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-worker-prod-dfw02
central-redis: argo deploy monitor/redisCluster.yaml deploymentGroup=mas-monitor-qa
agent-redis: argo deploy monitor/redisWorkerCluster.yaml deploymentGroup=mas-monitor-qa

You can deploy all application together by central and agent bundle. Here is production argo command for central and agent of MCT:

central: (one cluster version)
jfk: argo deploy monitor/central.yaml deploymentGroup=mas-monitor-prod-pri
dfw: argo deploy monitor/central.yaml deploymentGroup=mas-monitor-prod-gsb

central: (two cluster argo deploy version)
jfk&dfw: argo deploy monitor/central.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-prod-gsb

agent: (one cluster argo deploy version)
jfk: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-prod-pri
dfw: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-worker-prod-dfw02
ams: argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-worker-prod-ams02 (ams need ~, otherwise will fail)
fra: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-worker-prod-fra01
lhr: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-worker-prod-lhr05
jed: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-worker-prod-jed01
syd: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-worker-prod-syd01

agent: (two cluster argo deploy version)
dwf&ams: argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-worker-prod-dfw02|mas-monitor-worker-prod-ams02
jfk&fra: argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-worker-prod-fra01
lhr&jed: argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-worker-prod-lhr05|mas-monitor-worker-prod-jed01
syd: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-worker-prod-syd01

agent: (all cluster except gsb version)
argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-prod-pri|mas-monitor-worker-prod-fra01|mas-monitor-worker-prod-ams02|monitor-worker-prod-lhr05|mas-monitor-worker-prod-jed01|mas-monitor-worker-prod-syd01

Notice: agent GSB is mas-monitor-worker-prod-dfw02 instead of mas-monitor-prod-gsb and do not deploy monitor-worker on wdfwgen-p-4 (-gsb)

reason: wdfwgen-p-3 central has been migrated to wdfwgen-p-4, then mas-monitor-prod-gsb has been migrated from p-3 to p-4, but agent still in p-3 (dfw02)

Notice: mas-monitor-worker-prod-ruh01 is retired, not need to deploy

BTS

central: (one cluster version)
jfk: argo deploy monitor/central.yaml deploymentGroup=mas-monitor-bts-pri
dfw: argo deploy monitor/central.yaml deploymentGroup=mas-monitor-bts-gsb

central: (two cluster argo deploy version)
jfk&dfw: argo deploy monitor/central.yaml deploymentGroup=~mas-monitor-bts-pri|mas-monitor-bts-gsb

agent: (one cluster argo deploy version)
jfk: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-bts-pri
dfw: argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-bts-gsb

agent: (two cluster argo deploy version)
jfk&dfw: argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-bts-pri|mas-monitor-bts-gsb

QA

central:
argo deploy monitor/central.yaml deploymentGroup=mas-monitor-qa

agent:
argo deploy monitor/agent.yaml deploymentGroup=mas-monitor-qa

Replace "prod" to "bts" will be the BTS version, e.g. argo deploy monitor/api.yaml deploymentGroup=~mas-monitor-bts-pri|mas-monitor-bts-gsb

For QA, there is no GSB side, so argo command should be one side, e.g. argo deploy monitor/api.yaml deploymentGroup=mas-monitor-qa

Deployment is completed when the pull request github page argo-gen shows "Status: Succeeded".

Experience ¶

If you want to reduce the impact of Kubernetes deployment issues, you can deploy fewer items at a time. Here's why:

Sometimes, a single issue can cause the entire deployment to fail. For example, when deploying JFK Central and the agent together, an issue with one agent may cause problems for the central component, and then the central component may trigger more issues.
If you only deploy a few items, when an issue occurs, you can narrow down the scope of the problem and solve it more quickly.

Troubleshotting ¶

Argo deploy retry continue fail issue (no app issue) ¶

Compare the helm release list with the pod list, and uninstall the problematic helm releases:

The service in the release list is not in the deployed state.
There are differences between the release list and the pod list, for example, a service is missing in the release list while it exists in the pod list.

case:

argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-worker-prod-ruh01|mas-monitor-worker-prod-ams02
# continuous fail, then check service state with helm list:
helm list -n mas-monitor-worker
# if found mas-appium-runner is not deployed state in list, then uninstall it:
helm uninstall mas-appium-runner -n mas-monitor-worker

kubectl get deployments cmd can get more detailed deployed items

kubectl get deployments -n mas-monitor-worker

For example, helm list -n mas-monitor-worker will put some items in mas-micro-service like identity-management, but kubectl get deployments -n mas-monitor-worker will directly show identity-management..

Sometimes you may reinstall all item in a namespace to recover, e.g. no exception item or no use for uninstall exception ones:

helm uninstall -n mas-monitor-worker \
mas-appium-runner \
mas-common-identity-management \
mas-get-meeting-status \
mas-key-management \
mas-kube-job-management \
mas-meeting-security-key \
mas-message-service \
mas-nginx-rtmp \
mas-resource-management \
mas-schedule-meeting \
mas-start-meeting \
mas-worker-adapter-java \
mas-worker-adapter-python \
mas-worker-agent \
mas-worker-polling

in this case, sometime you may get "not found" issue:

lewan@LEWAN-M-6FV6 ~ % helm uninstall \
mas-appium-runner \
mas-common-identity-management \
mas-get-meeting-status \
mas-key-management \
mas-kube-job-management \
mas-meeting-security-key \
mas-message-service \
mas-nginx-rtmp \
mas-resource-management \
mas-schedule-meeting \
mas-start-meeting \
mas-worker-adapter-java \
mas-worker-adapter-python \
mas-worker-agent \
mas-worker-polling \
-n mas-monitor-worker
release "mas-appium-runner" uninstalled
Error: uninstall: Release not loaded: mas-common-identity-management: release: not found
lewan@LEWAN-M-6FV6 ~ % 

just ignore item before not found:
lewan@LEWAN-M-6FV6 ~ % helm uninstall \
mas-get-meeting-status \
mas-key-management \
mas-kube-job-management \
mas-meeting-security-key \
mas-message-service \
mas-nginx-rtmp \
mas-resource-management \
mas-schedule-meeting \
mas-start-meeting \
mas-worker-adapter-java \
mas-worker-adapter-python \
mas-worker-agent \
mas-worker-polling \
-n mas-monitor-worker
Error: uninstall: Release not loaded: mas-get-meeting-status: release: not found
lewan@LEWAN-M-6FV6 ~ % 

not found issue may happen again like above then go on this operation untill all item has been tried uninstalling

Argo Deployment Insufficient Resources Issue ¶

Sometimes, the pod appears to be healthy. However, when you click on the pod, you'll notice an insufficient resources issue.

Try using 7 CPU cores instead of 8.
Try using 4 CPU cores instead of 3.5.
Try setting maxUnavailable to 1 instead of 25%.
Ask the Kubed team to check if cluster nodes are available for use (sometimes nodes can't be scheduled). - You may need to reinstall the target deployment in this case

agent bundle deploy fail ¶

For example argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-worker-prod-ams02 fail with Insufficient cpu

helm uninstall -n mas-monitor-worker mas-appium-runner mas-worker-adapter-java mas-worker-agent mas-worker-polling mas-worker-adapter-python
redeploy with argo deploy monitor/agent.yaml deploymentGroup=~mas-monitor-worker-prod-ams02