1607744 - Intermittent toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_blocklist_telemetry.js | test_blocklist_useXML_scalar - [test_blocklist_useXML_scalar : 25] when the Firefox version increases to 75 on 2020-02-10

Status: VERIFIED FIXED

(Toolkit :: Add-ons Manager, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=283834554&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PK_KUCzZQq6jG8tKNcu7Nw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - "CONSOLE_MESSAGE: (info) 1578406612779 addons.manager DEBUG Completed startup sequence"
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - (xpcshell/head.js) | test run_next_test 1 pending (2)
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - (xpcshell/head.js) | test test_setup finished (2)
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_blocklist_telemetry.js | Starting test_blocklist_useXML_scalar
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - (xpcshell/head.js) | test test_blocklist_useXML_scalar pending (2)
[task 2020-01-07T14:16:53.135Z] 14:16:53 WARNING - TEST-UNEXPECTED-FAIL | toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_blocklist_telemetry.js | test_blocklist_useXML_scalar - [test_blocklist_useXML_scalar : 25] Got the expected value for blocklist.useXML scalar - "undefined" == false
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - Z:/task_1578403901/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_blocklist_telemetry.js:assertTelemetryScalars:25
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - Z:/task_1578403901/build/tests/xpcshell/tests/toolkit/mozapps/extensions/test/xpcshell/rs-blocklist/test_blocklist_telemetry.js:test_blocklist_useXML_scalar:46
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - Z:\task_1578403901\build\tests\xpcshell\head.js:run_next_test/_run_next_test/<:1567
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - Z:\task_1578403901\build\tests\xpcshell\head.js:_run_next_test:1567
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - Z:\task_1578403901\build\tests\xpcshell\head.js:run:735
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - Z:\task_1578403901\build\tests\xpcshell\head.js:_do_main:246
[task 2020-01-07T14:16:53.135Z] 14:16:53 INFO - Z:\task_1578403901\build\tests\xpcshell\head.js:_execute_test:573

Please check all the probes added by bug 1572711.

Comment 1

[Luca Greco [:rpl] [:luca] [:lgreco]]

Hi Gijs,
The telemetry probes we added in Bug 1572711 are currently set to expire in Firefox 75 (and so the tests fail on the beta simulation),
before I proceed to renew them I wanted to double-check with you first (if and for how many additional releases we do want to keep the probes related to the XML-based implementation).
e.g. do we want to renew all of them as they are for 2-3 more releases? what do you think?

Comment 2

[:Gijs (he/him)]

(In reply to Luca Greco [:rpl] [:luca] [:lgreco] from comment #1)

Hi Gijs,
The telemetry probes we added in Bug 1572711 are currently set to expire in Firefox 75 (and so the tests fail on the beta simulation),
before I proceed to renew them I wanted to double-check with you first (if and for how many additional releases we do want to keep the probes related to the XML-based implementation).
e.g. do we want to renew all of them as they are for 2-3 more releases? what do you think?

I think we should renew the XML ones for 2 more releases and the remote settings one for the habitual 6 months (which will be 6 or 7 releases). We'll want to monitor uptake and be able to compare the two probes, and we're only switching for the 72 release after the release, so having the XML probes gone on nightly at that point will be problematic if we have to stop the rollout or there are any other issues.

Comment 3

[:Gijs (he/him)]

Note that you'll need data review but there's a much shorter list of questions - see https://github.com/mozilla/data-review/blob/master/renewal_request.md .

Comment 4

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to :Gijs (he/him) from comment #2)

I think we should renew the XML ones for 2 more releases and the remote settings one for the habitual 6 months (which will be 6 or 7 releases). We'll want to monitor uptake and be able to compare the two probes, and we're only switching for the 72 release after the release, so having the XML probes gone on nightly at that point will be problematic if we have to stop the rollout or there are any other issues.

Thanks, that was what I was thinking of too (as I was pretty sure we still wanted the XML one for a bit more during the roll out, but likely for a smaller amount of releases than the RS one).

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p2 as the bug is tracked by a release manager for the current nightly.
See What Do You Triage for more information

Comment 6

[Intermittent Failures Robot]

1 failures in 4472 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1607744&startday=2020-01-06&endday=2020-01-12&tree=all

Comment 7

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: data-review-bug-1607744-blocklist-telemetry-renewal.md

Comment 8

[Luca Greco [:rpl] [:luca] [:lgreco]]

Attachment: Bug 1607744 - Extend telemetry probes related to the XML and RS blocklist. r?janerik!

Comment 9

[Chris H-C :chutten]

Comment on attachment 9120552 [details]
data-review-bug-1607744-blocklist-telemetry-renewal.md

Load-balancing to Nicole.

Comment 10

[Luca Greco [:rpl] [:luca] [:lgreco]]

Hi Nicole,
this issue is being tracked for Firefox 74 (because the expiring telemetry probes are going to trigger some test failures once we increase the version in mozilla-central) and so I was wondering if it would be possible for you to review this data request before the end of the week
(the data review form is the shorter one used for the renewals).

I'd like to land the patch (which is already signed off by a telemetry peer) before we are all going to travel to the All Hands next week.

Comment 11

[Nicole Shadowen]

Comment on attachment 9120552 [details]
data-review-bug-1607744-blocklist-telemetry-renewal.md

Thanks for your patience.

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions files [Scalars.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Scalars.yaml) and [Events.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Events.yaml) as well as in the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

No, probes related to the XML blocklist will expire in Firefox 77:

- Scalars.yaml:
  - blocklist.lastModified_xml
  - blocklist.useXML
- Events.yaml:
  - addonsManager.blocklistUpdateError 
 
Probes related to the RemoteSettings blocklist will expire in Firefox 81:

- Scalars.yaml:
  - blocklist.lastModified_rs_addons
  - blocklist.lastModified_rs_plugins


    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No.

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

Yes. Luca Greco is responsible for renewing or removing the collection before it expires in Firefox 77 for probes related to XML blocklist, and Firefox 81 for probes related to the RemoteSettings blocklist.

---
Result: datareview+

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/636a1dbf953c
Extend telemetry probes related to the XML and RS blocklist. r=janerik

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/636a1dbf953c

Comment 14

[Cosmin Sabou [:CosminS]]

Fix verified in today's version increase simulation: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedJob=287745908&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cusercancel%2Crunnable&revision=9601c58e8991f23fc2a442a37819feae731864f3&searchStr=xpc