Skip to content

How to sync data among regions in real time?

Background

GSS frontend is deployed on cloudFront, it is worldwide CDN, and GSS backend is under Route53, the primary side is in Virginia us-east-1, the secondary side is in Ohio us-east-2.

Currently the secondary side is just a backup region, which means the data is not synced between Virginia and Ohio realtime.

Only everyday at 7:00am UTC, a job would copy the Virginia RDS instance snapshot to Ohio, and then RDS in the Ohio would be restored from this snapshot, followed by rebuilding the Ohio backend service with the latest webApp packages.

Solution

But above architecture is not good enough for high availability, the backup region can not take over requests from the primary region due to the database restore delay. So a new architecture is required to solve this issue. In the new diagram, when there is a save/delete request to the backend service, meanwhile, a message would be sent through SNS(Simple Notification Service) to SQS (Simple Queue Service) across regions.

As following:

  1. When there is a new incident or incident message to be updated, tomcat instances in the primary region(us-east-1) would accept a URL request to handle.
  2. Once the data is stored successfully in the primary database, one record with the index is returned, then add it back to the incident or incident message objects.
  3. Some of the indexes are uuid, but others are number, to solve the conflict issue of the number index. the start value of number indexes are different among regions, but the step is the same. Now the start value is n in the primary, and n+1 in the secondary, the step is 3.
  4. One message with an object and its save/delete action is sent through SNS, then routed to SQS FIFO topics across regions such as us-east-2 or us-west-1.
  5. Tomcat instances in the other two regions would consume this FIFO topic and then execute exactly once. After that, the data replication across regions is completed.
  6. As two regions are in the active-standby mode, only the primary region would send out email notifications and RSS feeds.

Open event driven with cross region data replication (1).jpg syncdata.png