1615005 - Assist with SoftVision QA testing for sync migration

Status: RESOLVED FIXED

(Cloud Services :: Operations: Miscellaneous, task, P1)

Description

[Rachel Tublitz [:rachel]]

We have several softvision accounts that will be used for testing the migration script as part of PI-430

TIMING

Staging:
We're going to plan on starting on Feb 24th.

Production:
Date TBD

Account Info:

PRODUCTION ACCOUNTS
Firefox account Sync ID
[e-mail] 137900849
[e-mail] 137901151
[e-mail] 137901306
[e-mail] 137901574
[e-mail] 104435379
[e-mail] 135124277

STAGING ACCOUNTS
[e-mail] 5743942
[e-mail] 5743943
[e-mail] 5743944
[e-mail] 5743945
[e-mail] 5552941
[e-mail] 5743946

:jrconlin, ni?-ing you here for instructions on running the migration script against production and staging; can you detail them here?

Comment 1

[JR Conlin [:jrconlin,:jconlin]]

The migration script is located as part of the https://github.com/mozilla-services/syncstorage-rs repo under tools/user_migration.

Some instructions are provided in the README.md file displayed at https://github.com/mozilla-services/syncstorage-rs/tree/master/tools/user_migration

For stage migration, the mysql and spanner dsns should point to the stage instances. In addition, it may be useful to include the stage tokenserver database DSN as the --token_dsn argument. This will ensure that the tokenserver assigns the user to the spanner database after migration occurs.

A few additional notes:

Because of the complexities involved in sync and migration, the sync-storage server currently does NOT force the user to re-fetch a new token by sending a 4xx message at the completion of the migration. Sync across multiple devices tends to be a bit unpredictable. Instead, we rely on the client's token expiring, and the client electing to fetch a new token. This may take several hours, depending on the device. You may be able to hasten that by using a manual Refresh Token process. (Will follow up this comment with steps once I confirm with the sync team that the FAQ is still accurate)

The migration script has been tested using a Stand Alone server, since dev does not have access to stage or prod servers. Expect the unexpected, and bugs gleefully accepted.

Comment 2

[JR Conlin [:jrconlin,:jconlin]]

Following up from discussions with sync:

It may be possible to reset the sync token: "You can disconnect devices from the "devices and apps" list in account settings, which is similar to signing out of sync". This does also imply that logging out of sync and then logging back in should also suffice. There may be a desktop specific method to manually reset the sync token, however this will not be available to mobile devices. I will update as information becomes available.

Comment 3

[JR Conlin [:jrconlin,:jconlin]]

Further note from Sync:

"setting the pref services.sync.debug.ignoreCachedAuthCredentials [equal to true] will force the client to fetch a new token from the tokenserver every time it needs one"

This should force migration to fetch a new token, which would get the new assignment. It is recommended that you only set this AFTER the sync migration script completes to prevent accidental reassignment.

So the steps should be:

1. run the migration script for the user.
2. in about:config set services.sync.debug.ignoreCachedAuthCredentials to true.
3. force a sync via 'Sync Now`

This should cause the client to connect to the new sync store, and rectify.

Comment 4

[Bob Micheletto [:bobm]]

[e-mail] 5743945

This uid is now 5744115. The sync node it was allocated has been used for other testing. This id will need to be reconnected and re-sync before we perform any tests. I would like to migrate the user data over Wednesday, Feb. 26th, morning-time US/Pacific (or soon thereafter). However, for the testing to be most effective the sync clients should have some pre-populated data, and then disconnect and not sync again until after the data has been transferred and the users have been re-assigned.

Please verify that this will be possible.

Comment 5

[Stefan Deiac]

Hello Bob, I made some new unique browsing data and synced using the [e-mail] account and I disconnected the account after that.
I'll wait for your updates to start the verification part.

Comment 6

[Bob Micheletto [:bobm]]

(In reply to Stefan Deiac from comment #5)

I'll wait for your updates to start the verification part.

We are still working on the migration. I will provide further updates when we ready.

Comment 7

[Vasilica Tamas, QA [:vtamas]]

(In reply to Bob Micheletto [:bobm] from comment #4)

[e-mail] 5743945

This uid is now 5744115. The sync node it was allocated has been used for other testing. This id will need to be reconnected and re-sync before we perform any tests. I would like to migrate the user data over Wednesday, Feb. 26th, morning-time US/Pacific (or soon thereafter). However, for the testing to be most effective the sync clients should have some pre-populated data, and then disconnect and not sync again until after the data has been transferred and the users have been re-assigned.

Please verify that this will be possible.

We successfully imported all the sync data json files into the targeted users and we also created a few more for this testing:

[e-mail] 5469316
[e-mail] 5613640
[e-mail] 5744129
[e-mail] 5469295
[e-mail] 5469296

One of the testing objectives is to monitor the user behavior during migration which means that they do not need to be previously linked to spanner in order to analyze how each user behaves during the migration process. Besides the happy flow we have targeted also particular scenarios like:

• manual sync performed during migration
• Firefox account disconnection during migration
• closing the browser during migration
• network disconnection during migration

In conclusion, the testing users should remain assigned to MySQL and we need to be aware when the migration process will start.

Comment 8

[Bob Micheletto [:bobm]]

(In reply to Vasilica Tamas, QA from comment #7)

In conclusion, the testing users should remain assigned to MySQL and we need to be aware when the migration process will start.

I have set aside an hour to do the staging data migration on March 3rd, 10 AM US/Eastern. Please acknowledge if that time is acceptable. Let's coordinate the migration in the #services-engineering Slack channel.

Comment 9

[Vasilica Tamas, QA [:vtamas]]

(In reply to Bob Micheletto [:bobm] from comment #8)

(In reply to Vasilica Tamas, QA from comment #7)

In conclusion, the testing users should remain assigned to MySQL and we need to be aware when the migration process will start.

I have set aside an hour to do the staging data migration on March 3rd, 10 AM US/Eastern. Please acknowledge if that time is acceptable. Let's coordinate the migration in the #services-engineering Slack channel.

That sounds great. Thanks!

Comment 10

[Vasilica Tamas, QA [:vtamas]]

Thank you @bobm and @jconlin for this stage testing coordination!
Here are the testing results: https://testrail.stage.mozaws.net/index.php?/runs/view/26240&group_by=cases:section_id&group_order=asc

Comment 11

[Erik Olson]

I created a new node in production and assigned the prod users to that node = sync-node-815. This is staging for the prod user migration testing next week.

Comment 12

[Erik Olson]

Please enable those clients to sync their data to the new back end prior to testing next week.

Comment 13

[Erik Olson]

The new node is sync-815-us-west-2.sync.services.mozilla.com

Comment 14

[Rachel Tublitz [:rachel]]

Erik, it looks like we need two more users assigned for testing against production:

[e-mail] - 104435855
[e-mail] 135141393

Would you be able to assign these users as well?

Comment 15

[Erik Olson]

Yes, I will assign them to 815 shortly

Comment 16

[Erik Olson]

Those two users have been migrated to https://sync-815-us-west-2.sync.services.mozilla.com

Comment 17

[Rachel Tublitz [:rachel]]

Confirmed we're good to go with planning the next round of testing for tomorrow, March 11th starting at 7am PDT.

Comment 18

[Erik Olson]

We can meet here - https://mozilla.zoom.us/j/701352219

Comment 19

[Erik Olson]

Hello Stefan and Vasilica,
We fixed the abort script and migrated the [e-mail] user again, copying all data except bookmarks 3-10. Can you please test syncing that user now?