Closed Bug 1324167 Opened 7 years ago Closed 7 years ago

Create telemetry probes to track which preference categories are opened

Categories

(Firefox :: Settings UI, defect)

Product:
Firefox
Component:
Settings UI
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
VERIFIED FIXED
Milestone:
Firefox 53
Tracking Flags:
Tracking Status
firefox51 --- verified
firefox52 --- verified
firefox53 --- verified

People

(Reporter: jaws, Assigned: jaws)

References

Details

Attachments

(2 files)

Bug 1324167 - Create telemetry probes to track which preference categories are opened. privacy-review=bsmedberg
58 bytes, text/x-review-board-request
Gijs
: review+
Details
Patch for Aurora52 and Beta51
5.67 KB, patch
lizzard
: approval-mozilla-aurora+
lizzard
: approval-mozilla-beta+
Details | Diff | Splinter Review 
This is being requested as part of a preferences redesign and reorganization.
Feels like this would be more readable as a scalar. Do we have infrastructure to monitor that easily (like tmo's histogram tool) yet, Georg?
Flags: needinfo?(gfritzsche)
Comment on attachment 8819508 [details]
Bug 1324167 - Create telemetry probes to track which preference categories are opened.  privacy-review=bsmedberg

https://reviewboard.mozilla.org/r/99246/#review99590

r=me, but I think scalars would be better here if there's a reasonable way of reading the values afterwards. See below for some other comments.

::: browser/components/preferences/in-content/preferences.js:127
(Diff revision 2)
> +  switch (category) {
> +    case "general":

An alternative way to implement this would be to have an array and use `indexOf`. That would be more concise. Up to you if you think that reduces readability/flexibility too much.

::: browser/components/preferences/in-content/preferences.js:158
(Diff revision 2)
> +        case "encryptionTab":
> +          return 11;
> +      }
> +      // fall-through for default case.
> +    default:
> +      throw new Error(`Unexpected category in telemetryBucketForCategory: ${category}`);

Shouldn't this just put it in a fallback bucket (for which bucket 0 might be the most appropriate, actually) ?
Attachment #8819508 - Flags: review?(gijskruitbosch+bugs) → review+
(In reply to :Gijs Kruitbosch from comment #4)
> Feels like this would be more readable as a scalar. Do we have
> infrastructure to monitor that easily (like tmo's histogram tool) yet, Georg?

Scalars are already available in the longitudinal dataset.
They will be added to t.m.o in the near future (bug 1297063).

Note that support for recording scalars from content processes is only being added (bug 1278556).
Flags: needinfo?(gfritzsche)
Comment on attachment 8819508 [details]
Bug 1324167 - Create telemetry probes to track which preference categories are opened.  privacy-review=bsmedberg

https://reviewboard.mozilla.org/r/99246/#review99754

::: toolkit/components/telemetry/Histograms.json:5264
(Diff revision 2)
>    },
> +  "FX_PREFERENCES_CATEGORY_OPENED": {
> +    "bug_numbers": [1324167],
> +    "alert_emails": ["jaws@mozilla.com"],
> +    "expires_in_version": "56",
> +    "kind": "enumerated",

We have "kind":"categorical" now, which allows explicitly specifying labels.
See [choosing a histogram type](https://developer.mozilla.org/en-US/docs/Mozilla/Performance/Adding_a_new_Telemetry_probe#Choosing_a_Histogram_Type), `TELEMETRY_TEST_CATEGORICAL` is an example.
Hey Benjamin, can you do the telemetry/privacy review for this?

We would like to add an opt-out telemetry probe to track which preference categories users view. This will help us to know what categories are most popular, or at least to know what categories users try to visit when they have something in mind that they want to change.

These are set to expire in version 56.

We want an opt-out probe because we understand that users on Release channel use Firefox very differently than users on Nightly, especially when it comes to Preferences due to their "advanced" nature.

We may add some more probes in the future, which might track which preferences were changed and the time it takes from opening the preferences to the first preference changed, though that is not part of this bug or patch. We hope to use this data to help influence as well as measure the successfulness of reorganizing the preference categories as outlined in the specs on bug 1324168.
Flags: needinfo?(benjamin)
data-review=me
Flags: needinfo?(benjamin)
Pushed by jwein@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/551c356821b4
Create telemetry probes to track which preference categories are opened. r=Gijs privacy-review=bsmedberg
https://hg.mozilla.org/mozilla-central/rev/551c356821b4
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
status-firefox53: affected → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 53
Approval Request Comment
[Feature/Bug causing the regression]: telemetry probe to gather data about preference usage prior to reorg/redesign being tracked by bug 1324168
[User impact if declined]: none
[Is this code covered by automated tests?]: no, manual testing of telemetry probe was done locally
[Has the fix been verified in Nightly?]: not yet
[Needs manual test from QE? If yes, steps to reproduce]: no
[List of other uplifts needed for the feature/fix]: none
[Is the change risky?]: no
[Why is the change risky/not risky?]: adds a telemetry value when entering the preferences and another when changing categories
[String changes made/needed]: none
Attachment #8821191 - Flags: approval-mozilla-beta?
Attachment #8821191 - Flags: approval-mozilla-aurora?
Comment on attachment 8821191 [details] [diff] [review]
Patch for Aurora52 and Beta51

OK for aurora uplift. 
jaws, is there anything you can suggest for testing this over the next few days before we build the next beta?
Flags: needinfo?(jaws)
Attachment #8821191 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Andrei or Florin, it might be good to just check that enabling and disabling preferences works OK in aurora once this lands. I'm trying to think back on past issues with changing and adding new telemetry probes - usually performance problems or crashes.
This can still make it into the beta 11 build on Jan. 2.
Flags: needinfo?(florin.mezei)
Flags: needinfo?(andrei.vaida)
As an update, we are currently tracking an issue with this probe on telemetry.mozilla.org, bug 1326414. This doesn't appear to affect people using the preferences, and may be a sign of a larger issue with new telemetry probes.
(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #19)
> As an update, we are currently tracking an issue with this probe on
> telemetry.mozilla.org, bug 1326414. This doesn't appear to affect people
> using the preferences, and may be a sign of a larger issue with new
> telemetry probes.

The issue is not related to this probe but to the type of probe that this is using. This does not affect data collection or functionality of Firefox. It only affects viewing the data that was collected and will not require any changes to the probe.
Flags: needinfo?(jaws)
Comment on attachment 8821191 [details] [diff] [review]
Patch for Aurora52 and Beta51

Thanks jaws. OK, this can make it into the beta 12 build on Thursday.
Attachment #8821191 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
I reproduced this issue using 53.0a1, build ID: 20161216030207, on Windows 10 x64.
I can confirm this issue is fixed, I verified using Fx 51.0b12, build ID: 20170105155013 and Fx 52.0a2, build ID: 20170106004019 and Fx53.0a1, build ID: 20170106030204, on Windows 10 x64, Mac OS X 10.11 and Ubuntu 14.04 LTS.

Cheers!
Status: RESOLVED → VERIFIED
status-firefox51: fixed → verified
status-firefox52: fixed → verified
status-firefox53: fixed → verified
Flags: needinfo?(florin.mezei)
Flags: needinfo?(andrei.vaida)
Blocks: 1328561
No longer blocks: 1324168
Update from Frank Bertsch is that the category view on telemetry.mozilla.org is almost ready for viewing, but all data prior to Jan 17, 2017 was lost due to an issue with some processing code. I don't think that this data loss will cause a major change in our analysis, though we will want to wait a couple weeks longer before using the data to let more data collect. Frank says that we shouldn't expect any future data loss, this was a one-time issue and it was related to the new categorical type of probes.
Just want to point out that the data is *technically* still there, but histograms are all 0 (some other metadata isn't, so it truly messes up count information). I would recommend that you manually choose dates after 2017-01-17 in the distribution dashboard (under advanced options), and ignore all evolution data before that date as well (there's no way to limit the time range in the evolution view).
Blocks: 1330552
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: