1263835 - [meta] [SYNC-48] detect, measure and report inconsistent server bookmark states

Status: NEW

(Firefox :: Sync, defect, P3)

Description

[Mark Hammond [:markh] [:mhammond]]

Bug 1257970 calls for us to recover from inconsistent bookmark states. I feel that the best path to that is to firstly write some code that detects these states and report the results via telemetry.

Detecting this state could be a costly operation - it probably involves reading details about every bookmark on the server to turn it into a tree to be compared against the places version of the bookmarks tree - which could consume significant bandwidth and cause enough IO for Firefox to seem slow. So we may need to experiment on the best approach to take here in a relatively safe way - something like:

* land the code behind a nightly-only pref so we can analyze initial results before making it available to the wider population.

* Have some upper limit for the number of records on the server so we don't hit the few users with extreme bookmark counts (telemetry tells me that on release 44, the 95th percentile is 1250 bookmarks, 99.5 percentile is 8k bookmarks) and some conservative prefs for how often we perform this check and how we throttle.

* Write the actual comparison code (ie, the code that turns the server records into a tree, and the code that compares the local and remote trees)

* Record not only the results of the data comparison, but telemetry about the process itself (eg, how long did it take to fetch all records from the server, how often did we fail to get all the records, etc)

* Analyze the results and tweak the process such that we are confident we can make this available to a wider audience without negatively impacting the user. This might involve changing how we fetch the server data (eg, maybe we decide that instead of reading the records into memory we slowly create a mirror of the server locally and run our analysis against that.) Increase the audience to Aurora, Beta and finally release.

* Eventually use this same basic infrastructure to fix bug 1257970 (ie, make this process write corrected data to the server) and ensure the telemetry reflects that the corruption is indeed being fixed.

As a result, this bug is a meta bug - the actual work is likely to land as a number of separate bugs. We can make these bugs up as we go!