1488945 - browser_UsageTelemetry.js is going to permafail when the Gecko version number is bumped to 65 on 2018-10-22

Status: VERIFIED FIXED

(Firefox :: Tabbed Browser, defect)

Description

[Andreea Pavel [:apavel]]

[Tracking Requested - why for this release]:

+++ This bug was initially created as a clone of Bug #1425967 +++

[Tracking Requested - why for this release]: Permafailing tests when the Gecko version number is bumped to 60.

Treeherder link: https://treeherder.mozilla.org/#/jobs?repo=try&revision=701dd8f52ed8236eb93a1bf297ef81997a391170&selectedJob=197695277

Failure log:  https://treeherder.mozilla.org/logviewer.html#?job_id=197695277&repo=try&lineNumber=12227

[task 2018-09-05T21:23:29.622Z] 21:23:29     INFO - Leaving test bound test_subsessionSplit
[task 2018-09-05T21:23:29.624Z] 21:23:29     INFO - Entering test bound test_tabsHistogram
[task 2018-09-05T21:23:29.624Z] 21:23:29     INFO - Buffered messages finished
[task 2018-09-05T21:23:29.625Z] 21:23:29     INFO - TEST-UNEXPECTED-FAIL | browser/modules/test/browser/browser_UsageTelemetry.js | TAB_COUNT telemetry - initial tab counts - [794,173,2859] deepEqual [0,0,0] - JS frame :: chrome://mochitests/content/browser/browser/modules/test/browser/browser_UsageTelemetry.js :: checkTabCountHistogram :: line 141
[task 2018-09-05T21:23:29.626Z] 21:23:29     INFO - Stack trace:
[task 2018-09-05T21:23:29.627Z] 21:23:29     INFO - chrome://mochitests/content/browser/browser/modules/test/browser/browser_UsageTelemetry.js:checkTabCountHistogram:141
[task 2018-09-05T21:23:29.629Z] 21:23:29     INFO - chrome://mochitests/content/browser/browser/modules/test/browser/browser_UsageTelemetry.js:test_tabsHistogram:148
[task 2018-09-05T21:23:29.629Z] 21:23:29     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1102
[task 2018-09-05T21:23:29.630Z] 21:23:29     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1093
[task 2018-09-05T21:23:29.631Z] 21:23:29     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:995
[task 2018-09-05T21:23:29.632Z] 21:23:29     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:795
[task 2018-09-05T21:23:29.633Z] 21:23:29     INFO - GECKO(3192) | ++DOCSHELL 0xe6fa3c00 == 2 [pid = 3356] [id = {dbf38f89-3e73-43f3-8103-b9d3edac3c5b}]
[task 2018-09-05T21:23:29.635Z] 21:23:29     INFO - GECKO(3192) | ++DOMWINDOW == 9 (0xe6f30040) [pid = 3356] [serial = 69] [outer = (nil)]
[task 2018-09-05T21:23:29.636Z] 21:23:29     INFO - GECKO(3192) | ++DOMWINDOW == 10 (0xe6fa5800) [pid = 3356] [serial = 70] [outer = 0xe6f30040]
[task 2018-09-05T21:23:29.638Z] 21:23:29     INFO - GECKO(3192) | ++DOMWINDOW == 11 (0xe6fa6400) [pid = 3356] [serial = 71] [outer = 0xe6f30040]
[task 2018-09-05T21:23:29.641Z] 21:23:29     INFO - Not taking screenshot here: see the one that was previously logged
[task 2018-09-05T21:23:29.642Z] 21:23:29     INFO - TEST-UNEXPECTED-FAIL | browser/modules/test/browser/browser_UsageTelemetry.js | TAB_COUNT telemetry - opening tabs - [796,174,2868] deepEqual [0,0,1] - JS frame :: chrome://mochitests/content/browser/browser/modules/test/browser/browser_UsageTelemetry.js :: checkTabCountHistogram :: line 141
[task 2018-09-05T21:23:29.643Z] 21:23:29     INFO - Stack trace:
[task 2018-09-05T21:23:29.643Z] 21:23:29     INFO - chrome://mochitests/content/browser/browser/modules/test/browser/browser_UsageTelemetry.js:checkTabCountHistogram:141
[task 2018-09-05T21:23:29.644Z] 21:23:29     INFO - chrome://mochitests/content/browser/browser/modules/test/browser/browser_UsageTelemetry.js:test_tabsHistogram:153
[task 2018-09-05T21:23:29.645Z] 21:23:29     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1102
[task 2018-09-05T21:23:29.652Z] 21:23:29     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1093
[task 2018-09-05T21:23:29.652Z] 21:23:29     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:995
[task 2018-09-05T21:23:29.653Z] 21:23:29     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:795

TAB_COUNT histogram which is set to expire in firefox 65

Comment 1

[:Gijs (he/him)]

Stephen, can you confirm if we still want/need these statistics? If so, I need to extend the telemetry probes again.

Comment 2

[Stephen Horlander [:shorlander]]

(In reply to :Gijs (he/him) from comment #1)
> Stephen, can you confirm if we still want/need these statistics? If so, I
> need to extend the telemetry probes again.

Yes we would still like to have these statistics. Thanks :)

Comment 3

[:Gijs (he/him)]

Attachment: Bug 1488945 - make TAB_COUNT probes not expire, data-review,r?chutten

Comment 4

[:Gijs (he/him)]

(In reply to Stephen Horlander [:shorlander] from bug 1425967 comment #6)
> (In reply to :Gijs from comment #5)
> > FWIW, I think data/privacy review folks will want to have a little bit more
> > information about why this is important / what it gets us. Even something
> > like "we use this for every shield study to check retention / usage" or
> > "this helps us evaluate whether people use few/many tabs and how to
> > prioritize the UI design for those cases" or whatever. But "something" more
> > than just "we shouldn't let this expire. Especially if I'm flipping this to
> > 'never'. :-)
> 
> This "this helps us evaluate whether people use few/many tabs and how to
> prioritize the UI design for those cases" is right :)
> 
> Specifically in this case we have tentative plans for tab improvements in
> 2018 and having this data is likely going to be important.
> 
> On a more general note about all of these UI probes that track usage of
> things: they help us to make decisions on how to make Firefox better and
> what work to prioritize. The "issue" is that we don't always know when we
> are going to need the data. So if they expire then we can't retroactively
> get the information we need when we need it.


Patch up to never expire these probes based on the rationale provided by Stephen. Data review form:



What questions will you answer with this data?
 - how tab usage varies between users and/or over time.

Why does Mozilla need to answer these questions? Are there benefits for users? Do we need this information to address product or business requirements?

 - We need to be able to correlate product engineering/UX investment with how users use the product (e.g. tab hoarding vs. only a few tabs at a time), how that affects usage (are longer-time users also more-tabs users, or the opposite, or ...?) as well as gather data about effectiveness of such improvements (e.g. if we improve tab management, do people have more tabs open?).

What alternative methods did you consider to answer these questions? Why were they not sufficient?

 - Other methods were considered before this probe was implemented (in bug 1361855), but in practice we need to be able to monitor this data continuously and/or for specific projects. A permanent telemetry probe is the best way to do that.


List all proposed measurements and indicate the category of data collection for each measurement, using the Firefox data collection categories on the Mozilla wiki.

 - this is category 2 data.		

How long will this data be collected? Choose one of the following:

 - We want to permanently have this data available for UX purposes (on all channels/countries/locales).

If this data collection is default on, what is the opt-out mechanism for users?

 - standard telemetry metrics.

Please provide a general description of how you will analyze this data.

 - telemetry histogram viewer, custom sql or spark workbook queries for correlations and/or specific projects.

Where do you intend to share the results of your analysis?

 - publicly where appropriate, e.g. on the UX blog etc.

Comment 5

[:Gijs (he/him)]

(In reply to :Gijs (he/him) from comment #4)

:chutten, can you do the data-review here? We don't have a good way of doing this on phabricator, so not sure how best to ask, but there we are...

Comment 6

[Chris H-C :chutten|PTO (back Oct 23)]

This would be datareview- because never-expiring probes require someone in the alert_emails field and in the data collection request to be responsible (see request.md's question 6 answer 3 "I want to permanently monitor this data. (put someone’s name here)")

To not slow things down I'll assume that this will be Gijs so I can proceed with the review.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, standard Telemetry mechanisms apply.

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, standard Telemetry mechanisms apply.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

(see note above) Yes, :Gijs.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 2

    Is the data collection request for default-on or default-off?

Default on

    Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

No.

    Is the data collection covered by the existing Firefox privacy notice? If unsure: escalate to legal if:

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No, this is to be permanently collected.

---
Result: datareview+ (if Gijs or some other responsible person's email address ends up in the alert_emails and mentions themselves here as the one responsible for the probe)

Comment 7

[Chris H-C :chutten|PTO (back Oct 23)]

Comment on attachment 9010894 [details]
Bug 1488945 - make TAB_COUNT probes not expire, data-review,r?chutten

Chris H-C :chutten has approved the revision.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/80b7ae67aab7
make TAB_COUNT probes not expire, data-review+r=chutten

Comment 9

[Andrei Ciure[:aciure]]

https://hg.mozilla.org/mozilla-central/rev/80b7ae67aab7

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified fixed with the latest version increase simulation: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed,busted,exception,retry,usercancel,runnable&revision=65f11d59516f23b3edd12b8b5e993624bc149918