1180321 - [meta] Telemetry to help understand Fennec Sync and Firefox Account experience quality

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Android Sync, defect, P3)

Description

[Nick Alexander :nalexander [he/him]]

We're flying close to blind on the Fennec Sync and Firefox Account experience.  This ticket tracks adding some basic metrics to improve the situation.

Here are some ideas for data to collect:

Sync
----

* how frequently folks sync;
* how frequently folks sync specific collections;
* how long syncs take;
* how long specific collections take to sync;
* how many syncs succeed;
* how many syncs fail;
* how collections fail/succeed (local (database) error?  remote (network) error?);
* how many items get downloaded and uploaded;
* size of records download and uploaded.

* how many times we have to reconcile records (possible data loss);
* how many first syncs we see (since these are expensive).

Firefox Account metrics
-----------------------

* how many unverified accounts do we see?
* how long do users stay unverified?
* how often do we see password needed?
* how long do accounts live?
* do accounts get deleted frequently?

UI metrics
----------

* how do users sign up for accounts?  (First run?  From the synced tabs panel?  From the Android Settings menu?  Other?)
* how often do folks use the status activity?

Comment 1

[Nick Alexander :nalexander [he/him]]

I expect many sub-tickets, all of which are variations on a theme.  I want to mentor as much of this work as I can, so I'm going to centralize some of the getting started pieces here.

The telemetry system is an opt-in user data collection service run by Mozilla.  It is roughly documented at https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe.  We already have some migration telemetry in Sync.  I added that in Bug 1119061 -- it should be the model for all of these tickets.

There are 2 major parts to work like this:

1) adding histogram entries, like in https://hg.mozilla.org/mozilla-central/rev/94d2782392e4.
2) using the existing TelemetryWrapper and actually instrumenting the code in the right places.  See https://hg.mozilla.org/integration/fx-team/rev/72103138c055.

Comment 2

[Mark Hammond [:markh] [:mhammond]]

Thanks for filing this - Kit and I are working on the same basic bug for desktop.

One thing I'm not clear on here is what should be collected by the server vs the client.  Eg, most of the "Firefox Account metrics" items listed seem to be more suited for collection by the server, while for desktop we are focusing on things only the client knows (eg, how often do we lose the credentials?) or on things the server theoretically knows but will have trouble correlating with user experience (eg, are 401s from hawk closely related to users with a master-password or evenly distributed?)

Another interesting challenge will be to only measure "unexpected" occurrences - eg, "how many first syncs we see" is probably only valuable if we can exclude *actual* first syncs on a device - it's not yet clear to me if this needs to be done by the code inserting the probes, or if the new telemetry dashboards will be able to let us infer this (eg, a "first sync" that doesn't correlate with a "successfully setup an FxA on this profile" is probably not expected and of interest).

But having this bug is valuable - the answers to these questions should be roughly the same for Desktop as they are for Fennec, and even though the implementations may be quite different we should be able to agree on the semantics for many such probes.