Closed Bug 1595765 Opened 5 years ago Closed 5 years ago

Manual migration test for serveral durable sync users

Categories

(Cloud Services :: Operations: Miscellaneous, task, P2)

task

Tracking

(Not tracked)

RESOLVED INVALID

People

(Reporter: rachel, Unassigned)

Details

As the first step in testing out the implementation plan for the Durable Sync migration, we need some help on the Ops side to manually test this out for several sync users.

Here's what we need, for a few Sync ids (Rachel to add them here when ready):

  1. 503-inate the node in AWS (but don't kill the EC2 instance).
    • Is this even possible for one sync id? Do we need to stick a single sync id on it's own node as a proper test?
  2. Export data from EC2 instance, and import it into the GCP stack.
    • We'll want to talk through this together - specifically...what we don't have but need at the moment is:
      1. Details on the encryption method in existing sync - ie, is it tied to hostname/storage mechanism in a way that if it changes we'll have problems de-crypting in spanner?
      2. [MASSIVE TODO for Services Eng team] - Write the actual migration script. Even if it's just a quick prototype to start with, we need to test this as a script, not a manual process.
  3. Wait for 2 hours to ensure any active clients have given up on in-progress batch uploads.
    • 2 hours will be our first test. If we do well, we can (perhaps)? keep trying to get this time down. in the future
  4. Update Tokenserver db to re-allocate this sync node to GCP storage.
  5. Notify me when step 4 is complete so I can start testing right after it happens and document what I'm seeing.

The plan right now is to test this for a few users here, and once we feel comfortable, I'll write up documentation on the process, and we'll get a full QA cycle going here.

Rachel to attach sync id to use here before this can be started.

Flags: needinfo?(rtublitz)
Flags: needinfo?(rtublitz)
Priority: -- → P2

Sync id to use for testing: 130891406

Need to do this for a different account. Will attach when ready.

Rachel to include test migration script for use here before this can be started.

Flags: needinfo?(rtublitz)

(In reply to Rachel Tublitz [:tublitzed] from comment #0)

  1. 503-inate the node in AWS (but don't kill the EC2 instance).
    • Is this even possible for one sync id? Do we need to stick a single sync id on it's own node as a proper test?

Right now, we cannot easily do that. We could, for the purposes of a test, do it in Nginx. The method we discussed previously would be to modify the Sync server code to make this easier, and store migration state in memcached or in the DB.

:bobm - thanks, that's helpful. So, just to clarify here, if we were to store the per-user state somewhere (implementation being hashed out here, then would it be feasible on your end for us to have you make a manual change for a single user once we're ready? Assuming yes, once we have the ability to set/read user state flag, but just want to make sure that works on your end.

Flags: needinfo?(rtublitz)
Flags: needinfo?(bobm)
Summary: Manual migration needed as test for Durable Sync migration plan → Manual migration test for single durable sync user

Yes.

Flags: needinfo?(bobm)
Summary: Manual migration test for single durable sync user → Manual migration test for serveral durable sync users
Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → INVALID
You need to log in before you can comment on or make changes to this bug.