EMS Change Process ¶

Formal change process for EMS release planning, approval, execution, and rollback. Aligned with EMS 2.0 Change Management Service Epic.

1. Stages ¶

1.1 Create CHG in ServiceNow ¶

Every planned change must start with creating a CHG (Change Request) ticket in ServiceNow. The CHG serves as the single tracking record for all subsequent approval, execution, and post-change review activities.

Steps:

1. Log in to ServiceNow and create a Normal Change. For Example
2. Fill in the required fields:
   • Business service: Select the affected business service
   • Data center: Target data center(s) for the change
   • Short description: Brief summary of the change
   • Description: Detailed description of the change
3. Associate the CHG with the corresponding EMS release or feature ticket.
4. Submit the CHG into the approval workflow.

Gate: The CHG must receive all required approvals in ServiceNow (CAB or auto-approve, depending on risk level) before execution can begin.

1.2 Planning ¶

Release Lead ensures the following release artifacts are prepared:

Planned change window: Agree on deployment window that minimizes customer impact (avoid peak hours; coordinate with Change Management Service calendar).

1.3 Approval ¶

Required approvals before execution:

Gate: All approvals must be granted before proceeding to Communication stage.

1.4 Communication ¶

Before deployment:

1. Ask EMS/Ask AIOE team room notification — Post to Ask EMS/Ask AIOE with:
   • Change summary (what, when, expected duration)
   • Expected customer impact (if any)
   • Contact person for questions
   • Planned start and end time
2. Stakeholder notification — Notify downstream service teams if integration points are affected
3. Change Management Service — Update CR status to "Approved / Scheduled"

Template for Ask EMS / Ask AIOE notification:

📋 EMS Planned Change Notice
━━━━━━━━━━━━━━━━━━━━━━━━━━
Change: [SNOW Ticket]
Version: [x.x.x]
Window: [Start time] –> [End time] (UTC)
Down Time: [Yes/No]
Contact: [Release Lead name/alias]
Release Features:
- [Release feature list]
━━━━━━━━━━━━━━━━━━━━━━━━━━

1.5 Execution ¶

Follow the EMS Rollout Guide for technical steps:

1. Verify release branch and build package are ready
2. Update helm values (version, config, chart, secrets as needed)
3. Deploy to BTS first — validate before production
4. Deploy to Production (wjfk → wdfw04, one DC at a time)
5. Monitor deployment via Argo pipeline status

Special path for single-sided validation / primary-only feature testing:

Use this path when a feature cannot be fully validated in BTS and requires running on only one production side first, or when validation must occur on the current primary site.

1. Deploy to DFW first
2. Validate the feature on DFW while JFK remains on the previous version
3. If the feature can only be validated on the primary site, perform a controlled failover to move primary traffic to the upgraded side
4. Complete primary-site validation and confirm success criteria are met
5. Deploy the same version to JFK only after DFW validation succeeds
6. Perform a controlled failback if the standard primary topology needs to be restored

Gate for continuing past DFW: - Functional validation on DFW completed - No material customer impact observed - Failover validation completed if the feature requires primary-site testing - Release Lead and on-call agree production rollout can continue to JFK

During execution: - Deploy Engineer monitors deployment logs and health dashboards - On-call Engineer is available for immediate response - If deployment fails or health degrades → proceed to Rollback

1.6 Validation ¶

Post-deployment health checks (must all pass within 15 minutes of deploy):

Success criteria for lower customer impact: - Zero unplanned customer-facing alerts generated by the change itself - No increase in alert noise during change window - No escalation triggered by the planned change - Rollback (if needed) completed within defined SLA

Additional validation expectation for single-sided releases: - The non-upgraded side must remain stable while the first side is under test - Failover and failback must both be explicitly validated when used as part of the test plan - The second-side deployment must not start until the first-side validation record is captured in the change ticket or Ask EMS / Ask AIOE release thread

1.7 Rollback ¶

Trigger conditions (any one is sufficient): - Health checks fail after 15 minutes - Error rate exceeds 2× pre-deploy baseline - Customer-facing impact confirmed - Deploy Engineer / On-call judgment call

Rollback steps: 1. Revert helm values to previous version 2. Redeploy via Argo pipeline with previous package version 3. Confirm health checks pass with previous version 4. Notify Ask EMS / Ask AIOE team room: rollback performed, reason, next steps

Rollback SLA: Complete within 30 minutes of trigger decision.

1.8 Post-Change Review ¶

Within 2 business days of change completion:

1. Release Lead documents:
   • Was the change successful? (Yes / Partial / Rolled back)
   • Any unexpected issues encountered
   • Actual customer impact (if any)
   • Time from start to full validation
2. Lessons learned — What to improve for next release
3. Update Change Management Service — Close CR with outcome status
4. Ask EMS / Ask AIOE closure notification — Confirm change complete in team room

2. References ¶

• EMS Rollout Guide
• EMS Operation Runbook
• EMS Observability Architecture
• EMS Harness Deployment
• EMS Self Monitor