1513783 - Perma wpt-wr CCov [tier2] Error: chrome://marionette/content/PerTestCoverageUtils.jsm - Couldn't find target object for import.

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=216738963&repo=mozilla-central

https://queue.taskcluster.net/v1/task/HAOxLDZpSduHTUuOrFxEQg/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/HAOxLDZpSduHTUuOrFxEQg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2018-12-13T05:16:17.273Z] 05:16:17     INFO - PID 914 | Couldn't convert chrome URL: chrome://branding/locale/brand.properties
[task 2018-12-13T05:16:17.273Z] 05:16:17     INFO - PID 914 | [CodeCoverage] Setting handlers for process 1070.
[task 2018-12-13T05:16:17.536Z] 05:16:17    ERROR - Traceback (most recent call last):
[task 2018-12-13T05:16:17.536Z] 05:16:17    ERROR -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/executors/protocol.py", line 53, in setup
[task 2018-12-13T05:16:17.536Z] 05:16:17    ERROR -     getattr(self, cls.name).setup()
[task 2018-12-13T05:16:17.536Z] 05:16:17    ERROR -   File "/builds/worker/workspace/build/tests/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py", line 410, in setup
[task 2018-12-13T05:16:17.536Z] 05:16:17    ERROR -     self.is_enabled = self.marionette.execute_script(script)
[task 2018-12-13T05:16:17.536Z] 05:16:17    ERROR -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 1755, in execute_script
[task 2018-12-13T05:16:17.537Z] 05:16:17    ERROR -     rv = self._send_message("WebDriver:ExecuteScript", body, key="value")
[task 2018-12-13T05:16:17.538Z] 05:16:17    ERROR -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/decorators.py", line 26, in _
[task 2018-12-13T05:16:17.539Z] 05:16:17    ERROR -     return func(*args, **kwargs)
[task 2018-12-13T05:16:17.540Z] 05:16:17    ERROR -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 764, in _send_message
[task 2018-12-13T05:16:17.545Z] 05:16:17    ERROR -     self._handle_error(err)
[task 2018-12-13T05:16:17.545Z] 05:16:17    ERROR -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/marionette_driver/marionette.py", line 784, in _handle_error
[task 2018-12-13T05:16:17.546Z] 05:16:17    ERROR -     raise errors.lookup(error)(message, stacktrace=stacktrace)
[task 2018-12-13T05:16:17.546Z] 05:16:17    ERROR - JavascriptException: Error: chrome://marionette/content/PerTestCoverageUtils.jsm - Couldn't find target object for import.
[task 2018-12-13T05:16:17.548Z] 05:16:17    ERROR - stacktrace:
[task 2018-12-13T05:16:17.548Z] 05:16:17    ERROR - 	@tests/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py:411:7
[task 2018-12-13T05:16:17.548Z] 05:16:17    ERROR - 	@tests/web-platform/tests/tools/wptrunner/wptrunner/executors/executormarionette.py:410:2
[task 2018-12-13T05:16:17.548Z] 05:16:17    ERROR - 	evaluate.sandbox/promise<@chrome://marionette/content/evaluate.js:128:13
[task 2018-12-13T05:16:17.548Z] 05:16:17    ERROR - 	evaluate.sandbox@chrome://marionette/content/evaluate.js:97:17
[task 2018-12-13T05:16:17.548Z] 05:16:17    ERROR - 	GeckoDriver.prototype.execute_@chrome://marionette/content/driver.js:1029:19
[task 2018-12-13T05:16:17.549Z] 05:16:17    ERROR - 	async*GeckoDriver.prototype.executeScript@chrome://marionette/content/driver.js:899:24
[task 2018-12-13T05:16:17.549Z] 05:16:17    ERROR - 	Async*despatch@chrome://marionette/content/server.js:290:20
[task 2018-12-13T05:16:17.549Z] 05:16:17    ERROR - 	async*execute@chrome://marionette/content/server.js:263:11
[task 2018-12-13T05:16:17.550Z] 05:16:17    ERROR - 	async*onPacket/<@chrome://marionette/content/server.js:236:15
[task 2018-12-13T05:16:17.550Z] 05:16:17    ERROR - 	async*onPacket@chrome://marionette/content/server.js:235:8
[task 2018-12-13T05:16:17.551Z] 05:16:17    ERROR - 	_onJSONObjectReady/<@chrome://marionette/content/transport.js:493:9
[task 2018-12-13T05:16:17.551Z] 05:16:17    ERROR - 
[task 2018-12-13T05:16:17.551Z] 05:16:17    ERROR - 
[task 2018-12-13T05:16:17.588Z] 05:16:17     INFO - Browser exited with return code -15
[task 2018-12-13T05:16:17.591Z] 05:16:17     INFO - PROCESS LEAKS /tmp/tmpR0EJB5.mozrunner/runtests_leaks_856.log
[task 2018-12-13T05:16:17.591Z] 05:16:17     INFO - leakcheck | Processing log file /tmp/tmpR0EJB5.mozrunner/runtests_leaks_856.log for scope /2dcontext/building-paths
[task 2018-12-13T05:16:17.591Z] 05:16:17     INFO - TEST-INFO | leakcheck | default process: leak threshold set at 0 bytes
[task 2018-12-13T05:16:17.591Z] 05:16:17     INFO - TEST-INFO | leakcheck | plugin process: leak threshold set at 0 bytes
[task 2018-12-13T05:16:17.592Z] 05:16:17     INFO - TEST-INFO | leakcheck | tab process: leak threshold set at 10000 bytes
[task 2018-12-13T05:16:17.592Z] 05:16:17     INFO - TEST-INFO | leakcheck | geckomediaplugin process: leak threshold set at 20000 bytes
[task 2018-12-13T05:16:17.592Z] 05:16:17     INFO - TEST-INFO | leakcheck | gpu process: leak threshold set at 0 bytes
[task 2018-12-13T05:16:17.593Z] 05:16:17     INFO - TEST-INFO | leakcheck | rdd process: leak threshold set at 400 bytes
[task 2018-12-13T05:16:17.593Z] 05:16:17     INFO - leakcheck | Processing leak log file /tmp/tmpR0EJB5.mozrunner/runtests_leaks_856.log
[task 2018-12-13T05:16:17.593Z] 05:16:17     INFO - TEST-UNEXPECTED-FAIL | leakcheck | default missing output line for total leaks!
[task 2018-12-13T05:16:17.593Z] 05:16:17     INFO - leakcheck | Processing leak log file /tmp/tmpR0EJB5.mozrunner/runtests_leaks_856_tab_pid1024.log
[task 2018-12-13T05:16:17.593Z] 05:16:17     INFO - TEST-UNEXPECTED-FAIL | leakcheck | tab missing output line for total leaks!
[task 2018-12-13T05:16:17.594Z] 05:16:17     INFO - leakcheck | Processing leak log file /tmp/tmpR0EJB5.mozrunner/runtests_leaks_856_tab_pid997.log
[task 2018-12-13T05:16:17.594Z] 05:16:17     INFO - TEST-UNEXPECTED-FAIL | leakcheck | tab missing output line for total leaks!
[task 2018-12-13T05:16:17.595Z] 05:16:17     INFO - leakcheck | Processing leak log file /tmp/tmpR0EJB5.mozrunner/runtests_leaks_856_tab_pid1070.log
[task 2018-12-13T05:16:17.595Z] 05:16:17     INFO - TEST-UNEXPECTED-FAIL | leakcheck | tab missing output line for total leaks!

Comment 1

[Cosmin Sabou [:CosminS]]

The failures started after this merge to central: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=1e7843022450c852030788360f3a314de6678dac

Looked over the stack trace to find some clues but to no avail. One long shot would be https://hg.mozilla.org/mozilla-central/rev/1da261b7f019 
Aryx, please take a look over this. Thank you.

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Likely fallout from bug 1512816 and ccov wpt runs are broken due to this.

Comment 3

[James Graham [:jgraham]]

I don't *think* it should be that. I think ChromeUtils.import("chrome://marionette/content/PerTestCoverageUtils.jsm"); is failing, and I wonder if something happened that changed the URL of that file without wpt being updated.

Comment 4

[Marco Castelluccio [:marco]]

Is it possible to backfill to know when this started failing?

I haven't changed anything recently that could cause failure in importing PerTestCoverageUtils.jsm.

Comment 5

[Intermittent Failures Robot]

252 failures in 454 pushes (0.555 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-central: 252

Platform breakdown:
* linux64-ccov: 127
* windows10-64-ccov: 125

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-13&endday=2018-12-13&tree=all

Comment 7

[Marco Castelluccio [:marco]]

What is the first build that started failing?

Comment 8

[Andreea Pavel [:apavel]]

Marco: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=linux%2Cx64%2Cccov%2Cdebug%2Cweb%2Cplatform%2Ctests%2Cwith%2Ce10s%2Ctest-linux64-ccov%2Fdebug-web-platform-tests-reftests-e10s-3%2Cw-e10s%28wr3%29&tochange=1e7843022450c852030788360f3a314de6678dac&fromchange=88d304c633b6091a21fc64290256ef3fb51b7421&selectedJob=216690605

https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=1e7843022450c852030788360f3a314de6678dac

the first failure was on the push where this was filed, retriggers below are green.

Comment 9

[Marco Castelluccio [:marco]]

Could bug 1513277 have caused this?

Comment 10

[Jan de Mooij [:jandem]]

(In reply to Marco Castelluccio [:marco] from comment #9)
> Could bug 1513277 have caused this?

Yeah. It means there's a ChromeUtils.import(...) (or similar) of PerTestCoverageUtils.jsm where you need to pass |this| as second argument :)

Comment 11

[Andreea Pavel [:apavel]]

Jan, can you work on this?

Comment 12

[Kris Maglione [:kmag]]

I think maybe we want to fix this case by giving the Sandbox that stuff executes scripts in an appropriate ChromeUtils object for its realm... Should just need to pass `wantGlobalProperties: ["ChromeUtils"]` when creating it.

Comment 13

[Intermittent Failures Robot]

112 failures in 504 pushes (0.222 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* mozilla-central: 112

Platform breakdown:
* linux64-ccov: 59
* windows10-64-ccov: 50
* android-em-7-0-x86: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-14&endday=2018-12-14&tree=all

Comment 15

[Jan de Mooij [:jandem]]

I can try to write a patch for this tomorrow.

Comment 16

[Intermittent Failures Robot]

367 failures in 2485 pushes (0.148 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 367

Platform breakdown:
* linux64-ccov: 188
* android-em-7-0-x86: 3
* windows10-64-ccov: 176

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-10&endday=2018-12-16&tree=all

Comment 18

[Jan de Mooij [:jandem]]

Attachment: Bug 1513783 - Pass |this| explicitly to PerTestCoverageUtils.jsm imports to fix CCov tests. r?kmag

Comment 19

[Jan de Mooij [:jandem]]

(In reply to Kris Maglione [:kmag] from comment #12)
> I think maybe we want to fix this case by giving the Sandbox that stuff
> executes scripts in an appropriate ChromeUtils object for its realm...
> Should just need to pass `wantGlobalProperties: ["ChromeUtils"]` when
> creating it.

I don't know how or where to do this. The patch I submitted fixes this on Try though.

Comment 20

[Intermittent Failures Robot]

182 failures in 476 pushes (0.382 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 182

Platform breakdown:
* android-em-7-0-x86: 4
* windows10-64-ccov: 78
* linux64-ccov: 100

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-17&endday=2018-12-17&tree=all

Comment 21

[Jan de Mooij [:jandem]]

Attachment: Bug 1513783 - Give Marionette sandbox its own ChromeUtils object. r?kmag

Comment 22

[Intermittent Failures Robot]

148 failures in 505 pushes (0.293 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 147
* mozilla-inbound: 1

Platform breakdown:
* windows10-64-ccov: 87
* windows10-64: 1
* linux64-ccov: 60

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-18&endday=2018-12-18&tree=all

Comment 23

[Intermittent Failures Robot]

127 failures in 479 pushes (0.265 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. ** 

Repository breakdown:
* try: 1
* autoland: 1
* mozilla-central: 125

Platform breakdown:
* linux64-ccov: 125
* osx-10-10: 1
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-19&endday=2018-12-19&tree=all

Comment 24

[Intermittent Failures Robot]

229 failures in 441 pushes (0.519 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 229

Platform breakdown:
* windows10-64-ccov: 27
* android-em-7-0-x86: 1
* linux64-ccov: 200
* linux32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-20&endday=2018-12-20&tree=all

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/132871bdcd98
Give Marionette sandbox its own ChromeUtils object. r=kmag

Comment 26

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/132871bdcd98

Comment 27

[Intermittent Failures Robot]

101 failures in 416 pushes (0.243 failures/push) were associated with this bug yesterday.

Repository breakdown:
* mozilla-central: 100
* autoland: 1

Platform breakdown:
* linux64-ccov: 50
* linux64: 1
* windows10-64-ccov: 50

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-21&endday=2018-12-21&tree=all

Comment 28

[Intermittent Failures Robot]

787 failures in 2648 pushes (0.297 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. ** 

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. ** 

Repository breakdown:
* mozilla-central: 783
* try: 1
* mozilla-inbound: 1
* autoland: 2

Platform breakdown:
* windows10-64: 1
* linux64-ccov: 535
* windows10-64-ccov: 242
* linux32: 1
* android-em-7-0-x86: 5
* linux64: 1
* osx-10-10: 1
* windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-17&endday=2018-12-23&tree=all

Comment 29

[Intermittent Failures Robot]

4 failures in 1008 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 1
* autoland: 2
* mozilla-central: 1

Platform breakdown:
* osx-10-10: 1
* linux64: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1513783&startday=2018-12-24&endday=2018-12-30&tree=all