1623103 - Perma TEST-UNEXPECTED-FAIL | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | test_telemetry - [test_telemetry : 329] Extension left running at when version number gets increased to 77 on 2020-04-06

Status: RESOLVED FIXED

(WebExtensions :: General, defect, P1)

Description

[Cristian Brindusan [:cbrindusan]]

Central as Version Increase simulation:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=293507533&resultStatus=testfailed%2Cbusted%2Cexception&revision=b6c5aecb707dafe97258c42aff0dc001b6eeee4a

Failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=293507533&repo=try&lineNumber=4197

[task 2020-03-17T14:48:10.676Z] 14:48:10 INFO - TEST-START | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js
[task 2020-03-17T14:48:21.748Z] 14:48:21 WARNING - TEST-UNEXPECTED-FAIL | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | xpcshell return code: 0
[task 2020-03-17T14:48:21.748Z] 14:48:21 INFO - TEST-INFO took 11074ms
[task 2020-03-17T14:48:21.748Z] 14:48:21 INFO - >>>>>>>
[task 2020-03-17T14:48:21.748Z] 14:48:21 INFO - (xpcshell/head.js) | test MAIN run_test pending (1)
[task 2020-03-17T14:48:21.748Z] 14:48:21 INFO - (xpcshell/head.js) | test run_next_test 0 pending (2)
[task 2020-03-17T14:48:21.748Z] 14:48:21 INFO - (xpcshell/head.js) | test MAIN run_test finished (2)
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - running event loop
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | Starting check_remote
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - (xpcshell/head.js) | test check_remote pending (2)
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - TEST-PASS | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | check_remote - [check_remote : 1] useRemoteWebExtensions matches - true == true
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - TEST-PASS | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | check_remote - [check_remote : 1] testing from extension process - false == false
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - (xpcshell/head.js) | test run_next_test 0 finished (2)
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - (xpcshell/head.js) | test run_next_test 1 pending (2)
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - (xpcshell/head.js) | test check_remote finished (2)
[task 2020-03-17T14:48:21.749Z] 14:48:21 INFO - xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | Starting test_telemetry
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - (xpcshell/head.js) | test test_telemetry pending (2)
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - "Extension attached"
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - "Extension attached"
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - TEST-PASS | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | test_telemetry - [test_telemetry : 52] No data recorded for histogram: WEBEXT_CONTENT_SCRIPT_INJECTION_MS. - true == true
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - TEST-PASS | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | test_telemetry - [test_telemetry : 56] No data recorded for keyed histogram: WEBEXT_CONTENT_SCRIPT_INJECTION_MS_BY_ADDONID. - true == true
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - (xpcshell/head.js) | test run_next_test 1 finished (2)
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - "Started extension with id {a9462cdf-c498-fc4e-8d3e-3d2fa998f1bb}"
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - TEST-PASS | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | test_telemetry - [test_telemetry : 66] No data recorded for histogram after startup: WEBEXT_CONTENT_SCRIPT_INJECTION_MS. - true == true
[task 2020-03-17T14:48:21.750Z] 14:48:21 INFO - TEST-PASS | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | test_telemetry - [test_telemetry : 70] No data recorded for keyed histogram: WEBEXT_CONTENT_SCRIPT_INJECTION_MS_BY_ADDONID. - true == true
[task 2020-03-17T14:48:21.751Z] 14:48:21 INFO - Unexpected exception undefined - timed out after 50 tries.
[task 2020-03-17T14:48:21.751Z] 14:48:21 INFO - undefined
[task 2020-03-17T14:48:21.751Z] 14:48:21 INFO - exiting test
[task 2020-03-17T14:48:21.751Z] 14:48:21 WARNING - TEST-UNEXPECTED-FAIL | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_extension_content_telemetry.js | test_telemetry - [test_telemetry : 329] Extension left running at test shutdown - "running" == "unloaded"

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

Hi Cristian,
I'm wondering if there is a reason why you needinfo-ed mconley on this test failure that I may be missing.

This is very likely a failure due to some of the telemetry probes related to the WebExtensions framework that are currently set to expire on Firefox 77, and so the failure in the simulation is currently expected and I'm going to handle it by extending the probes we want to keep and remove the ones that are not being used (e.g. Bug 1621742).

Feel free to remove the needinfo-ed assigned to mconley and redirect it to me, if I'm right and Bug 1489524 isn't actually related to this.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Bug 1489524 has Mike as the patch author, that's why he got needinfoed.

Comment 4

[Luca Greco [:rpl] [:luca] [:lgreco]]

(In reply to Sebastian Hengst [:aryx] [limited availability until end of March](needinfo on intermittent or backout) from comment #3)

Bug 1489524 has Mike as the patch author, that's why he got needinfoed.

I'm pretty sure that Bug 1489524 is unrelated to this test failure in the Firefox 77 version increase simulation.
The test case that is failing is testing two histograms:

• WEBEXT_CONTENT_SCRIPT_INJECTION_MS, which mconley extended in Bug 1489524 to never expire: https://hg.mozilla.org/mozilla-central/rev/45e629e8498c#l1.564
• WEBEXT_CONTENT_SCRIPT_INJECTION_MS_BY_ADDONID, which is currently set to expire in Firefox 77: https://searchfox.org/mozilla-central/rev/d6f957415cf009995ecb539ef1425316d82164a9/toolkit/components/telemetry/Histograms.json#14024

I have also verified locally that extending WEBEXT_CONTENT_SCRIPT_INJECTION_MS_BY_ADDONID (to not expire in Firefox 77) does fix this failure as I was expecting (in a local build that does also include the changes needed to run the same kind of version increase simulation from the push to try linked in Comment 0).

I'm redirecting the needinfo from mconley to me (and I'm replacing Bug 1489524 with Bug 1578225 in the "regressed by" filed, as it is the bug from which we extended these probes near the end of last year)

Comment 5

[Luca Greco [:rpl] [:luca] [:lgreco]]

I just filed Bug 1623315 (to extend or remove the probes that are expiring in Firefox 77 and triggering this test failure) and assigned it to me.

Comment 6

[Ethan Tseng [:ethan]]

Luca, do you have any updates on this bug?

Comment 7

[Luca Greco [:rpl] [:luca] [:lgreco]]

Bug 1623315 has been merged to mozilla-central a couple of days ago, and so this failure shouldn't happen in the next "version increase" simulation.

I haven't closed this issue explicitly to let it be explicitly verified by a new "version increase" simulation.

Comment 8

[Alexandru Michis [:malexandru]]

Luca, sadly there are still a couple of failures on today's version increase:
https://treeherder.mozilla.org/#/jobs?repo=try&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=977e1b0ec819e44708c47989529e8fd44e846771&searchStr=%28X&selectedJob=295586654

Comment 9

[Luca Greco [:rpl] [:luca] [:lgreco]]

I took a look to those failures byt they looks unrelated to this issue, which was only tracking the failures due to "extension"-related probes that were expiring in Firefox 77.

I looked into a couple of them (test_select_default.js and test_select_named.js) and they seem to be failing because "startup.profile_selection_reason" scalar defined in Scalars.yaml is still marked as expiring in Firefox 77

I didn't looked all those tests in detail but my guess is that all the xpcshell tests in toolkit/profile/xpcshell that are failing in that push may be failing for the same reason and they should all be fixed by Bug 1623406.

Comment 10

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thank you for the explanation.