Skip to content

Mop for Data Sync New Solution

Precondition

  1. Make sure the California region is available [done] 1. Create CodePipeline for California-frontend (by cloudformation) 2. Deployment web and service by Jenkins pipeline 3. Cloudfront origin points to S3 on us-west-1 region 4. Health check for service: - https://service-status-us-west-1.webex.com/ping - https://service-status-us-west-1.webex.com/customer/dashServices/891 5. Check if it works: https://us-west-1.globalservicestatus.com

  2. Make sure that the resource of California region is consistent with the primary(db, redis, instance scale strategy) [done]

  3. Make sure API to change sequence start number is available on production(will deploy related api to production in advance) [done]
  4. Configuration 1. Create sns and sqs event for three regions, and subscribe to the sns topic of another region for the sqs queue of the two regions 2. Make sure that the parameters of data sync are added to AWS|Parameter Store for three regions: 1. /gss/prod/gss-service.amazon.sqs.dbEvent-queue-name 2. /gss/prod/gss-service.amazon.sns.topicArn 3. /gss/prod/gss-service.amazon.dataSync.sameRegion 4. /gss/prod/gss-service.amazon.dataSync.enable 5. /gss/prod/gss-service.amazon.kms.subscription-email-multi-region-key 3. Update the environment parameter from stage to prod for backup(refer to: GSS backup environment update)[done] 1. Check all parameters for prod folder 2. Change to prod in Beanstalk|Configuration 3. Update S3 folder in CodePipeline
  5. Create muti-region kms key in us-east-1, and replica kms key in us-east-2 and us-west-1
  6. add SNS/SQS access for prod ebInstance role

Note: Remember to list the sequence number to be set for each region two days before going production.

current_id set_id(us-east-1) set_id(us-west-1) set_id(us-east-2)
status_maintenance_event 5217 5300 5301 5302
status_maintenance_rss 265 300 301 302
status_maintenance_clusters 30 40 41 42
status_maintenance_dcs 10 20 21 22
status_release_note 28 30 31 32
status_incident_subscription_info 37068 38000 38001 38002
status_announcements_subscription_info 2821 3000 3001 3002
status_api_doc 11 20 21 22

Mop

  1. Notify customer about the maintenance window, this window would be at least from step 2 to 8
  2. Trigger db daily sync to make sure that the datas of all tables are synchronized(by modifying the snapshot taking time on AWS|RDS|Backup)
  3. Close daily sync by disable RDS|Event Subscription
  4. Change /gss/prod/spring.datasource.url to the value of /gss/stage/spring.datasource.url on California|Parameter Store
  5. Delete the replication rules on AWS|S3|gss-cisco-dist-bucket|Management
  6. Set sequence start number for the following tables by calling api to Virginia and California and Ohio 1. status_maintenance_event (PUT +/status/maintenance/event/sequence/{id}) 2. status_maintenance_rss (PUT +/status/maintenance/announcement/sequence/{id}) 3. status_maintenance_clusters (PUT +/status/cluster/sequence/{id}) 4. status_maintenance_dcs (PUT +/status/dataCenter/sequence/{id}) 5. status_release_note (PUT +/status/releaseNote/sequence/{id}) 6. status_incident_subscription_info (PUT +/subscription/incident/sequence/{id}) 7. status_announcements_subscription_info (PUT +/subscription/announcement/sequence/{id}) 8. status_api_doc (PUT +/status/apiDoc/sequence/{id})
  7. Deploy frontend Virginia
  8. Deploy backend Virginia
  9. Check the current region: https://service-status.webex.com/ping
  10. Health check for service: https://service-status.webex.com/customer/dashServices/891
  11. Check if it works: https://us-east-1.globalservicestatus.com and https://status.webex.com
  12. Deploy frontend California
  13. Deploy backend California
  14. Health check for service: https://service-status-us-west-1.webex.com/ping and https://service-status-us-west-1.webex.com/customer/dashServices/891
  15. Check if it works: https://us-west-1.globalservicestatus.com
  16. Config the secondary of service.globalservicestatus.com to gss-micr-884GT9H17JYE.eba-am9p7mmf.us-west-1.elasticbeanstalk.com for backend on Route53|Hosted zones|globalservicestatus.com
  17. Copy the latest rss and json files from status.webex.com.s3b to us-west-1-status.webex.com.s3b on AWS|S3
  18. Config failover for E3NG7SVV048ZOS(for status.webex.com) on AWS|Cloudfront 1. primary: status.webex.com.s3b 2. secondary: us-west-1-status.webex.com.s3b
  19. Create an invalidation for E3NG7SVV048ZOS(for status.webex.com) and E2SQZXY5DR6KHB(for us-west-1.globalservicestatus.com) on AWS|Cloudfront
  20. Check again if it works: https://us-west-1.globalservicestatus.com and https://status.webex.com

Ohio deployment

  1. Change /gss/prod/spring.datasource.url to the value of /gss/stage/spring.datasource.url on Ohio|Parameter Store
  2. Update the environment parameter from stage to prod for Ohio|Beanstalk|Configuration|Software|Environment properties and update S3 folder in backend-CodePipeline
  3. Deploy frontend Ohio
  4. Deploy backend Ohio
  5. Health check for service: https://service-status-us-east-2.webex.com/ping and https://service-status-us-east-2.webex.com/customer/dashServices/891
  6. Check if it works: https://us-east-2.globalservicestatus.com
  7. Copy the latest rss and json files from status.webex.com.s3b to us-east-2-status.webex.com.s3b on AWS|S3
  8. Create an invalidation for ERICE3EEZVEAA(for us-east-2.globalservicestatus.com) on AWS|Cloudfront