1578262 - Perma browser_telemetry_completeBadSize.js | The update.session.from_app_version value should exist when Gecko version increases to 72 on 2019-10-21

Status: VERIFIED FIXED

(Toolkit :: Application Update, defect, P2)

Description

[Raul Gurzau (:RaulG)]

[Tracking Requested - why for this release]:

Version increase simulation:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=264557176&resultStatus=testfailed%2Cbusted%2Cexception&revision=b8832c913b1c7150d74456e803a49ebe41d376f2

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=264557176&repo=try&lineNumber=5895

Log snippet:
[task 2019-09-02T12:20:13.160Z] 12:20:13 INFO - TEST-START | toolkit/mozapps/update/tests/browser/browser_telemetry_completeBadSize.js
[task 2019-09-02T12:20:13.223Z] 12:20:13 INFO - TEST-INFO | started process screentopng
[task 2019-09-02T12:20:13.738Z] 12:20:13 INFO - TEST-INFO | screentopng: exit 0
[task 2019-09-02T12:20:13.739Z] 12:20:13 INFO - Buffered messages logged at 12:20:13
[task 2019-09-02T12:20:13.740Z] 12:20:13 INFO - Entering test bound setupTestCommon
[task 2019-09-02T12:20:13.740Z] 12:20:13 INFO - Leaving test bound setupTestCommon
[task 2019-09-02T12:20:13.740Z] 12:20:13 INFO - Entering test bound telemetry_completeBadSize
[task 2019-09-02T12:20:13.740Z] 12:20:13 INFO - Buffered messages finished
[task 2019-09-02T12:20:13.741Z] 12:20:13 INFO - TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/browser/browser_telemetry_completeBadSize.js | The update.session.from_app_version value should exist. - false == true - JS frame :: chrome://mochitests/content/browser/toolkit/mozapps/update/tests/browser/head.js :: checkTelemetryUpdatePhases :: line 1331
[task 2019-09-02T12:20:13.741Z] 12:20:13 INFO - Stack trace:
[task 2019-09-02T12:20:13.741Z] 12:20:13 INFO - chrome://mochitests/content/browser/toolkit/mozapps/update/tests/browser/head.js:checkTelemetryUpdatePhases:1331
[task 2019-09-02T12:20:13.741Z] 12:20:13 INFO - chrome://mochitests/content/browser/toolkit/mozapps/update/tests/browser/browser_telemetry_completeBadSize.js:telemetry_completeBadSize:19
[task 2019-09-02T12:20:13.742Z] 12:20:13 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1350
[task 2019-09-02T12:20:13.742Z] 12:20:13 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1385
[task 2019-09-02T12:20:13.743Z] 12:20:13 INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1213
[task 2019-09-02T12:20:13.744Z] 12:20:13 INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:805
[task 2019-09-02T12:20:13.744Z] 12:20:13 INFO - Not taking screenshot here: see the one that was previously logged
[task 2019-09-02T12:20:13.745Z] 12:20:13 INFO - TEST-UNEXPECTED-FAIL | toolkit/mozapps/update/tests/browser/browser_telemetry_completeBadSize.js | The update.session.from_app_version value should equal the expected value. - "undefined" == "72.0a1" - JS frame :: chrome://mochitests/content/browser/toolkit/mozapps/update/tests/browser/head.js :: checkTelemetryUpdatePhases :: line 1332
[task 2019-09-02T12:20:13.746Z] 12:20:13 INFO - Stack trace:
[task 2019-09-02T12:20:13.746Z] 12:20:13 INFO - chrome://mochitests/content/browser/toolkit/mozapps/update/tests/browser/head.js:checkTelemetryUpdatePhases:1332
[task 2019-09-02T12:20:13.747Z] 12:20:13 INFO - chrome://mochitests/content/browser/toolkit/mozapps/update/tests/browser/browser_telemetry_completeBadSize.js:telemetry_completeBadSize:19
[task 2019-09-02T12:20:13.747Z] 12:20:13 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest/<:1350
[task 2019-09-02T12:20:13.747Z] 12:20:13 INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1385
[task 2019-09-02T12:20:13.747Z] 12:20:13 INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1213
[task 2019-09-02T12:20:13.747Z] 12:20:13 INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<:805
[task 2019-09-02T12:20:13.747Z] 12:20:13 INFO - TEST-PASS | toolkit/mozapps/update/tests/browser/browser_telemetry_completeBadSize.js | The update.session.mar_partial_size_bytes value should not exist. - true == true -
[task 2019-09-02T12:20:13.747Z] 12:20:13 INFO - Not taking screenshot here: see the one that was previously logged

Comment 1

[Robert Strong (they/them - no direct email)]

Attachment: Bug 1578262 - extend expiration from 72 to 76 for telemetry used to analyze BITS. r=chutten

Comment 2

[Robert Strong (they/them - no direct email)]

Attachment: data collection review

Comment 3

[Robert Strong (they/them - no direct email)]

Attachment: data_review.csv

Comment 4

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9090755 [details]
data collection review

Thanks!

1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in Scalars.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Type 1, technical data

5) Is the data collection request for default-on or default-off?

Default-on

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

rstrong is responsible for renewing as necessary.

9) Does the data collection use a third-party collection tool?

Nope

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/624d7ade4c15
extend expiration from 72 to 76 for telemetry used to analyze BITS. r=chutten

Comment 6

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/624d7ade4c15

Comment 7

[Natalia Csoregi [:nataliaCs]]

Verified with today's beta-sim: https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=0511424b95c084ee467d0062b774a88b6d518750

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Verified fixed with latest version increase simulation: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=b30d003a9e626cc9ace7f954c0a5eebbb2d5c9cd