1558111 - Intermittent devtools/client/webreplay/mochitest/browser_dbg_rr_breakpoints-01.js | Uncaught exception - at chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/helpers/context.js:11 - TypeError: panel is undefined

Status: RESOLVED FIXED

(Core Graveyard :: Web Replay, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=250871971&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/MvyF58uIQ8aLoqhYQUHfKQ/runs/0/artifacts/public/logs/live_backing.log

---

04:34:00 INFO - TEST-START | devtools/client/webreplay/mochitest/browser_dbg_rr_breakpoints-01.js
04:34:00 INFO - GECKO(1709) | MIDDLEMAN 1715 /var/folders/q5/387x1tqd2pngd0ldgl0y5xp4000017/T/TempRecording.1709.1
04:34:00 INFO - GECKO(1709) | RECORDING 1716 /var/folders/q5/387x1tqd2pngd0ldgl0y5xp4000017/T/TempRecording.1709.1
04:34:02 INFO - GECKO(1709) | REPLAYING 1718 /var/folders/q5/387x1tqd2pngd0ldgl0y5xp4000017/T/TempRecording.1709.1
04:34:02 INFO - GECKO(1709) | REPLAYING 1719 /var/folders/q5/387x1tqd2pngd0ldgl0y5xp4000017/T/TempRecording.1709.1
04:34:02 INFO - GECKO(1709) | REPLAYING 1720 /var/folders/q5/387x1tqd2pngd0ldgl0y5xp4000017/T/TempRecording.1709.1
04:34:02 INFO - GECKO(1709) | REPLAYING 1721 /var/folders/q5/387x1tqd2pngd0ldgl0y5xp4000017/T/TempRecording.1709.1
04:34:27 INFO - GECKO(1709) | 2019-06-10 04:34:27.074 firefox[1709:20652] Persistent UI failed to open file file:///Users/cltbld/Library/Saved%20Application%20State/org.mozilla.nightly.savedState/window_1.data: No such file or directory (2)
04:34:37 INFO - GECKO(1709) | ***** Fatal Record/Replay Error *****
04:34:37 INFO - GECKO(1709) | Hung replaying process
04:34:37 INFO - GECKO(1709) | console.error: "Exception while opening the toolbox" "[object Object]" ({error:"connectionClosed", message:"'resume' active request packet to 'server1.conn0.child1/context18' can't be sent as the connection just closed.", from:(void 0)})
04:34:37 INFO - GECKO(1709) | undefined
04:34:37 INFO - TEST-INFO | started process screencapture
04:34:37 INFO - TEST-INFO | screencapture: exit 0
04:34:37 INFO - Buffered messages logged at 04:34:00
04:34:37 INFO - Entering test bound
04:34:37 INFO - Waiting for event: 'RecordingFinished' on [object ParentProcessMessageManager].
04:34:37 INFO - Buffered messages logged at 04:34:01
04:34:37 INFO - Console message: [JavaScript Error: "The character encoding of the HTML document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the page must be declared in the document or in the transfer protocol." {file: "http://example.com/browser/devtools/client/webreplay/mochitest/examples/doc_rr_basic.html" line: 0}]
04:34:37 INFO - Got event: 'RecordingFinished' on [object ParentProcessMessageManager].
04:34:37 INFO - Buffered messages finished
04:34:37 INFO - TEST-UNEXPECTED-FAIL | devtools/client/webreplay/mochitest/browser_dbg_rr_breakpoints-01.js | Uncaught exception - at chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/helpers/context.js:11 - TypeError: panel is undefined
04:34:37 INFO - Stack trace:
04:34:37 INFO - createDebuggerContext@chrome://mochitests/content/browser/devtools/client/debugger/test/mochitest/helpers/context.js:11:15
04:34:37 INFO - attachRecordingDebugger@chrome://mochitests/content/browser/devtools/client/webreplay/mochitest/head.js:40:15
04:34:37 INFO - async*@chrome://mochitests/content/browser/devtools/client/webreplay/mochitest/browser_dbg_rr_breakpoints-01.js:11:21
04:34:37 INFO - Tester_execTest/<@chrome://mochikit/content/browser-test.js:1115:34
04:34:37 INFO - Tester_execTest@chrome://mochikit/content/browser-test.js:1143:12
04:34:37 INFO - nextTest/<@chrome://mochikit/content/browser-test.js:1004:14
04:34:37 INFO - SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:803:59
04:34:37 INFO - Leaving test bound
04:34:37 INFO - Console message: [JavaScript Error: "remote browser crashed while on http://example.com/browser/devtools/client/webreplay/mochitest/examples/doc_rr_basic.html
04:34:37 INFO - " {file: "chrome://mochikit/content/mochitest-e10s-utils.js" line: 8}]
04:34:37 INFO - e10s_init/<@chrome://mochikit/content/mochitest-e10s-utils.js:8:8
04:34:37 INFO - EventListener.handleEvente10s_init@chrome://mochikit/content/mochitest-e10s-utils.js:6:10
04:34:37 INFO - testInit@chrome://mochikit/content/browser-test.js:105:5
04:34:37 INFO - setTimeout handler@chrome://mochikit/content/browser-test.js:23:1
04:34:37 INFO - loadChromeScripts@jar:file:///var/folders/q5/387x1tqd2pngd0ldgl0y5xp4000017/T/tmpEUSIy3.mozrunner/extensions/[e-mail]!/api.js:13:25
04:34:37 INFO - loadMochitest@jar:file:///var/folders/q5/387x1tqd2pngd0ldgl0y5xp4000017/T/tmpEUSIy3.mozrunner/extensions/[e-mail]!/api.js:112:3
04:34:37 INFO - @tests/mochitest/runtests.py:1783:5
04:34:37 INFO - @tests/mochitest/runtests.py:1784:8
04:34:37 INFO - evaluate.sandbox/promise<@chrome://marionette/content/evaluate.js:128:10
04:34:37 INFO - evaluate.sandbox@chrome://marionette/content/evaluate.js:106:17
04:34:37 INFO - GeckoDriver.prototype.execute_@chrome://marionette/content/driver.js:1029:28
04:34:37 INFO - async*GeckoDriver.prototype.executeScript@chrome://marionette/content/driver.js:911:29
04:34:37 INFO - despatch@chrome://marionette/content/server.js:289:40
04:34:37 INFO - execute@chrome://marionette/content/server.js:262:16
04:34:37 INFO - onPacket/<@chrome://marionette/content/server.js:235:20
04:34:37 INFO - onPacket@chrome://marionette/content/server.js:236:9
04:34:37 INFO - _onJSONObjectReady/<@chrome://marionette/content/transport.js:492:20
04:34:37 INFO -
04:34:37 INFO - Removing tab.
04:34:37 INFO - Waiting for event: 'TabClose' on [object XULElement].
04:34:37 INFO - Got event: 'TabClose' on [object XULElement].
04:34:37 INFO - Tab removed and finished closing
04:34:37 INFO - GECKO(1709) | JavaScript error: resource://devtools/client/framework/toolbox.js, line 484: TypeError: this.threadClient is undefined
04:34:37 INFO - Not taking screenshot here: see the one that was previously logged
04:34:37 INFO - TEST-UNEXPECTED-FAIL | devtools/client/webreplay/mochitest/browser_dbg_rr_breakpoints-01.js | A promise chain failed to handle a rejection: this.threadClient is undefined - stack: _stopThreadClientListeners@resource://devtools/client/framework/toolbox.js:484:5

Comment 1

[Andreea Pavel [:apavel]]

Hi Joel.
The dt-wr seems to have a very high failure rate, however, the tests that are failing differ. Most of them fail with the issue here: Uncaught exception - at <test_path> - TypeError: panel is undefined

Here is a glimpse: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&group_state=expanded&searchStr=macosx1014-64-shippable%2Copt%2Cmochitests%2Ctest-macosx1014-64-shippable%2Fopt-mochitest-devtools-webreplay-e10s%2Cm%28dt-wr%29&fromchange=fd26384a1def16b578df46754799cde0fb7ee230&selectedJob=250871971

Could you take a look and tell us how to move forward with this?

Thank you.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

can we bisect this back to a root cause with retriggers? The failure rate is high enough that this should be tier-3 or try only until it is stable.

Comment 3

[Brian Hackett [Laid off!]]

I believe this and the other web replay test failures that have cropped up in the past several days are due to bug 1533675. Revision 7fc2d7c0c634 adds a call to NS_SetCurrentThreadName that happens at process startup, before record/replay state is initialized. This causes any locks which are lazily created within NS_SetCurrentThreadName to not be recorded, leading to deadlocks later on.

Comment 4

[Brian Hackett [Laid off!]]

Attachment: Bug 1558111 - Initialize record/replay state first in XRE_InitChildProcess, r=ehsan.

Comment 5

[Andreea Pavel [:apavel]]

First round of retriggers: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&group_state=expanded&searchStr=macosx1014-64-shippable%2Copt%2Cmochitests%2Ctest-macosx1014-64-shippable%2Fopt-mochitest-devtools-webreplay-e10s%2Cm%28dt-wr%29&fromchange=061cca57657dbdcdd8a31ee58229dd92fca1be48

Second round shows that Brian's intuition is right: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&group_state=expanded&searchStr=macosx1014-64-shippable%2Copt%2Cmochitests%2Ctest-macosx1014-64-shippable%2Fopt-mochitest-devtools-webreplay-e10s%2Cm%28dt-wr%29&tochange=061cca57657dbdcdd8a31ee58229dd92fca1be48&fromchange=967c30311690aa759c57271883dadb641e47db4b&selectedJob=250948001

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/78a3f864177f
Initialize record/replay state first in XRE_InitChildProcess, r=ehsan.

Comment 7

[Brian Hackett [Laid off!]]

Thanks for confirming, Andreea!

Comment 8

[Tarek Ziadé (:tarek)]

Thanks for the fix Brian

Comment 9

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/78a3f864177f

Comment 10

[Intermittent Failures Robot]

2 failures in 4753 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• macosx1014-64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1558111&startday=2019-06-10&endday=2019-06-16&tree=all