EMS Rollout Guide ¶

Create Release brunch ¶

Sample:
https://sqbu-github.cisco.com/Monitoring/keephq/tree/release/2.1.5

Build Packages ¶

After release brunch created, git will auto create build packages job in Jenkins and trigger run, when it's running, watch the jenkins console log page and will need manually input the version number.
jenkins build job

Modify Release Notes ¶

https://sqbu-github.cisco.com/Monitoring/docs/blob/main/docs/event-manager/release.md

Deploy Package ¶

Modify the Secret if needed: ¶

base on the app-secrets, modify the secret value on vault

Prod secret path:
APP: secret/prod/alert/prod/ems
DB: secret/prod/alert/prod/wdfw and secret/prod/alert/prod/wjfk
Redis: secret/prod/alert/prod/redis and secret/prod/alert/prod/ems: ems.redis
CI: secret/prod/alert/prod/ems: ems.ci

BTS secret path:
APP: secret/prod/alert/bts/ems
DB: secret/prod/alert/bts/wdfw and secret/prod/alert/bts/wjfk
Redis: secret/prod/alert/bts/redis and secret/prod/alert/bts/ems: ems.redis
CI: secret/prod/alert/bts/ems: ems.ci

Commit pr to deply ¶

create brunch support empty pr for app-config repo:
git commit -m "deploy micro service" --allow-empty

Create a empty pr for deploy: ¶

Sample:
https://sqbu-github.cisco.com/Monitoring/kubed-app-configs/pull/1450

After approved, add comment in pr like "argo deploy alert/ems.yaml deploymentGroup=xxxx" to trigger argo deploy:
Stage:

argo deploy alert/ems.yaml deploymentGroup=~mas-alert-bts-wdfw|mas-alert-bts-wjfk

Prod:

argo deploy alert/ems.yaml deploymentGroup=mas-alert-prod-wjfk
argo deploy alert/ems.yaml deploymentGroup=mas-alert-prod-wdfw04

EMS Secret Rotate Guide ¶

DB Secret Rotate ¶

Login to DB server, apply new PG DB account and grant same permission as old account.

# ─────────────────────────────────────────────
# STEP 1: Login to psql as superuser
# ─────────────────────────────────────────────
psql -U postgres

# ─────────────────────────────────────────────
# STEP 2: Create new user with same permission as old user (replace {new_username} and {new_password} with actual value)
# ─────────────────────────────────────────────
CREATE USER {new_username} WITH SUPERUSER NOCREATEDB NOCREATEROLE INHERIT LOGIN REPLICATION;
ALTER USER {new_username} WITH PASSWORD '{new_password}';

# ─────────────────────────────────────────────
# STEP 3: Verify the new user was created correctly
# ─────────────────────────────────────────────
psql -U {new_username} -d {DB_name} -c "SELECT 1;"

Update the secret value in vault, see Modify the Secret section for secret path.
Redeploy EMS service to make new secret take effect, see Commit pr to deply section for deploy steps.
After new account works well, revoke old account permission and delete it. Be careful to make sure the old account is not used by data sync job before revoke and delete it.

Redis Secret Rotate ¶

Update the redis cluster pwd value in vault, see Modify the Secret section for secret path. In App path, update the redis_password value for app to connect to redis, in Redis path, update the password value to set the new password.
Uninstall Redis cluster with helm command after login to kubernetes cluster:
helm uninstall status-redis-cluster -n mas-alert-<prod|bts>
Install Redis cluster with argo pipeline, see Commit pr to deply and run argo command like below:
argo deploy alert/status-redisCluster.yaml deploymentGroup=<deploymentGroup>
deploymentGroup: mas-alert-bts-wdfw, mas-alert-bts-wjfk, mas-alert-prod-wjfk, mas-alert-prod-wdfw04
Redeploy EMS service to make new secret take effect, see Commit pr to deply section for deploy steps.

LDAP Secret Rotate ¶

Go to adam.cisco.com, create new LDAP account.
Update the LDAP account password in vault, see Modify the Secret section for secret path. Update the ldap_password value in App path to make app can connect to LDAP server with new password.
Update the LDAP account name in app-config.
Redeploy EMS service to make new secret take effect, see Commit pr to deply section for deploy steps.
After new account works well, revoke old account permission and delete it.

CI Client Secret Rotate ¶

Apply new CI Client in SNOW-Identity OAuth Client with same Client Scopes and Redirection URIs as old client. you can get old client info from ci-ops-portal
Update the CI client id & secret in vault, see Modify the Secret section for secret path. Update the ci_client_id and ci_client_secret values.
Redeploy EMS service to make new secret take effect, see Commit pr to deply section for deploy steps.
After new client works well, request removal of old client in ServiceNow.

CI OAuth Resource Secret Rotate ¶

Apply new OAuth Resource in SNOW-Identity OAuth Service. you can get old resource info from ci-ops-portal
Update the OAuth resource id & secret in vault, see Modify the Secret section for secret path. Update the ci_service_id and ci_service_secret values.
Apply new scope for client id to fit with new OAuth Resource. Like {new_resource_name:read}
Redeploy EMS service to make new secret take effect, see Commit pr to deply section for deploy steps.
After new resource works well, request removal of old resource in ServiceNow.

API Key & PAT Rotate ¶

login EMS->click your name->settings->api keys / PAT ->click regenerate keys icon