1631787 - Please assist SoftVision with migration testing against production on Tuesday April 28th at 10am EST

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Operations: Sync, task, P1)

Description

[Rachel Tublitz [:rachel]]

Vasilica's team will be testing the migration as part of this PI request.

I'm marking this as a P1 as we're planning to start the migration on May 4th assuming all goes ok.

Erik and JR: can you be on hand to help troubleshoot? JR: could you also detail specifically what flags to use for the script?

Comment 1

[Erik Olson]

Yes, sounds great!

Comment 2

[JR Conlin [:jrconlin,:jconlin]]

There are now several scripts that will need to be run. The (README.md)[https://github.com/mozilla-services/syncstorage-rs/tree/master/tools/user_migration_ file] includes instructions about how to run these. All scripts support --help which will list all available options, their functions and default values (if appropriate). Most of the option default values have been suggested by operations, so no changes should be required. It is presumed that the scripts will have default read and write access to the current directory.

By default, the gen_bso_users.py --hoard_limit value does not have a default value. @eolson may have recommendations about what a good value to use is.

Note, these scripts will generate a success_{date}.log and failure_{date}.log file (if appropriate) indicating the successfully and unsuccessfully migrated userids.

Comment 3

[Vasilica Tamas, QA [:vtamas]]

Hi! Here are the selected accounts for this second part testing of sync data migration:

[e-mail] 128709160
[e-mail] 128754071
[e-mail] 128803402
[e-mail] 128803495
[e-mail] 128804907
[e-mail] 128806959
[e-mail] 128808758
[e-mail] 128809197
[e-mail] 128808946
[e-mail] 128809645
[e-mail] 128812632
[e-mail] 128812699

Comment 4

[Erik Olson]

Hello, they have been assigned to the node we will use for migration. Their new uids are as follows:

[e-mail] 128709160 [e-mail] 143683269
[e-mail] 128754071 [e-mail] 143683279
[e-mail] 128803402 [e-mail] 143683290
[e-mail] 128803495 [e-mail] 143683297
[e-mail] 128804907 [e-mail] 143683302
[e-mail] 128806959 [e-mail] 143683311
[e-mail] 128808758 [e-mail] 143683318
[e-mail] 128809197 [e-mail] 143683321
[e-mail] 128808946 [e-mail] 143683328
[e-mail] 128809645 [e-mail] 143683334
[e-mail] 128812632 [e-mail] 143683341
[e-mail] 128812699 [e-mail] 143683351

The node is up, please allow them to sync prior to the testing.

Comment 5

[Erik Olson]

I used 300k as a hoard limit in a recent test, that seemed reasonable as it would take about 4.5 minutes to transfer a user with that many rows and less than .5% of users on the node had more than that.

Comment 7

[Erik Olson]

Migration tests were completed yesterday. There were 12 clients tested. The beginning of the test was delayed due to the clients having sync errors while connecting to the migration sync-py node and deleting all their data. We have reproduced this issue today and it may need to be addressed.

The clients were re-populated and the migration to spanner testing was completed.

Attaching two clients logs showing the failures that led to data loss.

Comment 8

[Erik Olson]

Attachment: sync errors for 143683269.txt

Comment 9

[Erik Olson]

Attachment: sync errors for 143683279.txt

Comment 10

[Erik Olson]

New uids to get server logs after data migration to spanner:

[e-mail] 128709160 [e-mail] 144074002
[e-mail] 128754071 [e-mail] 144074484
[e-mail] 128803402 [e-mail] 144073987
[e-mail] 128803495 [e-mail] 144075243
[e-mail] 128804907 [e-mail] 144076549
[e-mail] 128806959 [e-mail] 144076072
[e-mail] 128808758 [e-mail] 144074443
[e-mail] 128809197 [e-mail] 144075229
[e-mail] 128808946 [e-mail] 144076550
[e-mail] 128809645 [e-mail] 144124338
[e-mail] 128812632 [e-mail] 144076546
[e-mail] 128812699 [e-mail] 144125169

Comment 11

[Erik Olson]

Attachment: 4-28-clients.csv.tgz

Comment 12

[Erik Olson]

Serverside logs are attached. 144074443, 144075243, and 144073987 re-uploaded their bookmarks. 144124338 and 144125169 have no activity.

Comment 13

[Philip Jenvey [:pjenvey]]

(In reply to Erik Olson from comment #7)

The beginning of the test was delayed due to the clients having sync errors while connecting to the migration sync-py node and deleting all their data. We have reproduced this issue today and it may need to be addressed.

The first log shows a hostname lookup failure (oddly enough this happens shortly after a successful GET to the same sync node).

The second log shows no /meta/global record. The is expected since this is their first connection to a sync-py node (https://sync-815-us-west-2.sync.services.mozilla.com) where they have no existing data. So the client performs a reset which includes deleting everything on the server:

1588075958472	Sync.Service	INFO	No metadata record, server wipe needed
1588075958472	Sync.Service	INFO	Wiping server data
1588075958472	Sync.Service	INFO	Fresh start. Resetting client.

The clients were re-populated and the migration to spanner testing was completed.

Attaching two clients logs showing the failures that led to data loss.

All I see is the delete against the sync-py node client logs (which isn't data loss if there was no data there in the first place).

Do we have the client logs against the spanner node?

Comment 14

[Philip Jenvey [:pjenvey]]

Vasilica can you make these client logs available?

Comment 15

[Vasilica Tamas, QA [:vtamas]]

Attachment: sync-error3.txt

I found the following sync errors, are these the right ones?

Comment 16

[Vasilica Tamas, QA [:vtamas]]

Attachment: sync-error4.txt

Comment 17

[Erik Olson]

I think we have determined the sync problems/data loss with node sync-815-us-west-2 are because of the workflow. Neither the client nor the server had the data, so there was nothing to sync.

Do you have client logs for 144074443, 144075243, and 144073987 as they re-uploaded their bookmarks to the spanner node, which they should not have done?

Comment 18

[Vasilica Tamas, QA [:vtamas]]

Here are the logs for the following users:

• 144075243:
  • https://drive.google.com/file/d/1iWibNI8RdXTa5LchZu8d5uFcs8lrf9ty/view?usp=sharing
• 144074443:
  • https://drive.google.com/file/d/1FhUAI7F4E-CmBSxq4Eij7ejbRiJ0__pW/view?usp=sharing
  • https://drive.google.com/file/d/1iAm6tFFaVoDIfgHyF8cGGOyAxlyFP8cF/view?usp=sharing
  • https://drive.google.com/file/d/1fSLBhx6AHuOE1ECtMHTxcj11rwroYi5m/view?usp=sharing

Comment 19

[Rebecca Billings [:rbillings]]

Verified sync users being migrated with correct history and bookmarks. A final round of manual testing is planned.