Closed Bug 1803419 Opened 2 years ago Closed 1 year ago

Intermittent Assertion failure: cx_->hadResourceExhaustion(), at /builds/worker/checkouts/gecko/js/src/jit/WarpOracle.cpp:192

Categories

(Core :: JavaScript Engine: JIT, defect, P5)

Product:
Core
Component:
JavaScript Engine: JIT
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
120 Branch
Tracking Flags:
Tracking Status
firefox-esr102 --- unaffected
firefox-esr115 --- unaffected
firefox117 --- unaffected
firefox118 --- wontfix
firefox119 --- fixed
firefox120 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: iain)

References

Details

(Keywords: assertion, intermittent-failure, Whiteboard: [retriggered])

Filed by: mlaza [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=398197687&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QlU-WFrxSamXIy1Iu_c5zg/runs/0/artifacts/public/logs/live_backing.log

[task 2022-11-30T18:28:54.270Z] 18:28:54     INFO - TEST-START | toolkit/mozapps/extensions/test/browser/browser_html_list_view_recommendations.js
[task 2022-11-30T18:28:54.298Z] 18:28:54     INFO - GECKO(1012) | [Child 2444: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 15d17931c00 == 2 [pid = 2444] [id = 115]
[task 2022-11-30T18:28:54.305Z] 18:28:54     INFO - GECKO(1012) | [Child 2444: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 3 (15d1277b260) [pid = 2444] [serial = 234] [outer = 0]
[task 2022-11-30T18:28:54.305Z] 18:28:54     INFO - GECKO(1012) | [Child 2444: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 4 (15d17932800) [pid = 2444] [serial = 235] [outer = 15d1277b260]
[task 2022-11-30T18:28:54.342Z] 18:28:54     INFO - GECKO(1012) | [Parent 6916: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 1fe3a9d4c00 == 15 [pid = 6916] [id = 240]
[task 2022-11-30T18:28:54.347Z] 18:28:54     INFO - GECKO(1012) | [Parent 6916: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 106 (1fe2f2707d0) [pid = 6916] [serial = 676] [outer = 0]
[task 2022-11-30T18:28:54.347Z] 18:28:54     INFO - GECKO(1012) | [Parent 6916: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 107 (1fe3a9d8400) [pid = 6916] [serial = 677] [outer = 1fe2f2707d0]
[task 2022-11-30T18:28:54.358Z] 18:28:54     INFO - GECKO(1012) | [Parent 6916: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 108 (1fe3a9d8800) [pid = 6916] [serial = 678] [outer = 1fe2f2707d0]
[task 2022-11-30T18:28:54.369Z] 18:28:54     INFO - GECKO(1012) | [Parent 6916, Main Thread] WARNING: Failed to retarget HTML data delivery to the parser thread.: file /builds/worker/checkouts/gecko/parser/html/nsHtml5StreamParser.cpp:1234
[task 2022-11-30T18:28:54.372Z] 18:28:54     INFO - GECKO(1012) | [Parent 6916, Main Thread] WARNING: 'aOwner->IsDiscarded()', file /builds/worker/workspace/obj-build/dist/include/mozilla/dom/SyncedContextInlines.h:95
[task 2022-11-30T18:28:54.473Z] 18:28:54     INFO - GECKO(1012) | [Parent 6916, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005 (NS_ERROR_FAILURE): file /builds/worker/checkouts/gecko/dom/base/nsContentUtils.cpp:4236
[task 2022-11-30T18:28:54.905Z] 18:28:54     INFO - GECKO(1012) | Assertion failure: cx_->hadResourceExhaustion(), at /builds/worker/checkouts/gecko/js/src/jit/WarpOracle.cpp:192
[task 2022-11-30T18:28:55.075Z] 18:28:55     INFO - GECKO(1012) | #01: js::jit::WarpOracle::createSnapshot() [js/src/jit/WarpOracle.cpp:191]
[task 2022-11-30T18:28:55.082Z] 18:28:55     INFO - GECKO(1012) | #02: js::jit::CreateWarpSnapshot(JSContext*, js::jit::MIRGenerator*, JS::Handle<JSScript *>) [js/src/jit/Ion.cpp:1527]
[task 2022-11-30T18:28:55.082Z] 18:28:55     INFO - GECKO(1012) | #03: js::jit::Compile(JSContext*, JS::Handle<JSScript *>, js::jit::BaselineFrame*, unsigned char*) [js/src/jit/Ion.cpp:1800]
[task 2022-11-30T18:28:55.083Z] 18:28:55     INFO - GECKO(1012) | #04: IonCompileScriptForBaseline(JSContext*, js::jit::BaselineFrame*, unsigned char*) [js/src/jit/Ion.cpp:2056]
[task 2022-11-30T18:28:55.084Z] 18:28:55     INFO - GECKO(1012) | #05: ??? (???:???)
[task 2022-11-30T18:28:55.336Z] 18:28:55     INFO - GECKO(1012) | [GPU 1476, IPC I/O Child] WARNING: [EAA7EED765A88535.5127FE81D8FAEAC8]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2022-11-30T18:28:55.346Z] 18:28:55     INFO - GECKO(1012) | [GPU 1476, IPC I/O Child] WARNING: [EAA7EED765A88535.5127FE81D8FAEAC8]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2022-11-30T18:28:55.346Z] 18:28:55     INFO - GECKO(1012) | [GPU 1476, IPC I/O Child] WARNING: [EAA7EED765A88535.5127FE81D8FAEAC8]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2022-11-30T18:28:55.346Z] 18:28:55     INFO - GECKO(1012) | [GPU 1476, IPC I/O Child] WARNING: [EAA7EED765A88535.5127FE81D8FAEAC8]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2022-11-30T18:28:55.346Z] 18:28:55     INFO - GECKO(1012) | [GPU 1476, Main Thread] WARNING: Shutting down GPU process early due to a crash!: file /builds/worker/checkouts/gecko/gfx/ipc/GPUParent.cpp:663
[task 2022-11-30T18:28:55.349Z] 18:28:55     INFO - GECKO(1012) | [GPU 1476, IPC I/O Child] WARNING: [EAA7EED765A88535.5127FE81D8FAEAC8]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2022-11-30T18:28:55.350Z] 18:28:55     INFO - GECKO(1012) | [GPU 1476, IPC I/O Child] WARNING: [EAA7EED765A88535.5127FE81D8FAEAC8]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2022-11-30T18:28:55.350Z] 18:28:55     INFO - GECKO(1012) | [Child 4592, IPC I/O Child] WARNING: [7C639DA1CECB4A16.376635515C86ACD0]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2022-11-30T18:28:55.352Z] 18:28:55     INFO - GECKO(1012) | [Child 4592, IPC I/O Child] WARNING: [7C639DA1CECB4A16.376635515C86ACD0]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2022-11-30T18:28:55.352Z] 18:28:55     INFO - GECKO(1012) | Exiting due to channel error.
[task 2022-11-30T18:28:55.439Z] 18:28:55     INFO - TEST-INFO | Main app process: exit 1
[task 2022-11-30T18:28:55.440Z] 18:28:55    ERROR - TEST-UNEXPECTED-FAIL | ShutdownLeaks | process() called before end of test suite
[task 2022-11-30T18:28:55.441Z] 18:28:55     INFO - TEST-INFO | Confirming we saw 489 DOCSHELL created and 464 destroyed log strings.
[task 2022-11-30T18:28:55.441Z] 18:28:55     INFO - TEST-INFO | Confirming we saw 1229 DOMWINDOW created and 1094 destroyed log strings.
[task 2022-11-30T18:28:55.442Z] 18:28:55     INFO - Buffered messages logged at 18:28:54
[task 2022-11-30T18:28:55.443Z] 18:28:55     INFO - Entering setup bound 
[task 2022-11-30T18:28:55.443Z] 18:28:55     INFO - Leaving setup bound 
[task 2022-11-30T18:28:55.444Z] 18:28:55     INFO - Entering test bound testExtensionList
[task 2022-11-30T18:28:55.444Z] 18:28:55     INFO - Loading manager window in tab
[task 2022-11-30T18:28:55.445Z] 18:28:55     INFO - TEST-PASS | toolkit/mozapps/extensions/test/browser/browser_html_list_view_recommendations.js | Should have an add-ons manager window -

Backfills so far.

Flags: needinfo?(bvandersloot)
Regressed by: 1782091
Flags: needinfo?(bvandersloot)
No longer regressed by: 1782091

There have been 30 total failures in the last 7 days, recent failure log.
Affected platforms are:

  • linux1804-64-qr
  • macosx1015-64-qr
  • windows10-32-2004-qr
  • windows10-64-2004-qr
  • windows11-32-2009-qr
  • windows11-64-2009-qr
[task 2023-03-06T00:49:57.530Z] 00:49:57     INFO - TEST-START | toolkit/mozapps/extensions/test/browser/browser_html_list_view_recommendations.js
[task 2023-03-06T00:49:57.560Z] 00:49:57     INFO - GECKO(6480) | [Child 4920: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 1497f482800 == 2 [pid = 4920] [id = 115]
[task 2023-03-06T00:49:57.561Z] 00:49:57     INFO - GECKO(6480) | [Child 4920: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 3 (1490414d280) [pid = 4920] [serial = 232] [outer = 0]
[task 2023-03-06T00:49:57.561Z] 00:49:57     INFO - GECKO(6480) | [Child 4920: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 4 (1497f483000) [pid = 4920] [serial = 233] [outer = 1490414d280]
[task 2023-03-06T00:49:57.599Z] 00:49:57     INFO - GECKO(6480) | [Parent 5064: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 1cc8e8a6000 == 15 [pid = 5064] [id = 242]
[task 2023-03-06T00:49:57.601Z] 00:49:57     INFO - GECKO(6480) | [Parent 5064: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 176 (1cc97e4d5e0) [pid = 5064] [serial = 682] [outer = 0]
[task 2023-03-06T00:49:57.602Z] 00:49:57     INFO - GECKO(6480) | [Parent 5064: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 177 (1cc952a4400) [pid = 5064] [serial = 683] [outer = 1cc97e4d5e0]
[task 2023-03-06T00:49:57.613Z] 00:49:57     INFO - GECKO(6480) | [Parent 5064: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 178 (1cc97e65800) [pid = 5064] [serial = 684] [outer = 1cc97e4d5e0]
[task 2023-03-06T00:49:57.624Z] 00:49:57     INFO - GECKO(6480) | [Parent 5064, Main Thread] WARNING: Failed to retarget HTML data delivery to the parser thread.: file /builds/worker/checkouts/gecko/parser/html/nsHtml5StreamParser.cpp:1235
[task 2023-03-06T00:49:57.627Z] 00:49:57     INFO - GECKO(6480) | [Parent 5064, Main Thread] WARNING: 'aOwner->IsDiscarded()', file /builds/worker/workspace/obj-build/dist/include/mozilla/dom/SyncedContextInlines.h:95
[task 2023-03-06T00:49:59.348Z] 00:49:59     INFO - GECKO(6480) | [Parent 5064, Main Thread] WARNING: '!top', file /builds/worker/checkouts/gecko/dom/xul/MenuBarListener.cpp:99
[task 2023-03-06T00:49:59.492Z] 00:49:59     INFO - GECKO(6480) | [Parent 5064, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005 (NS_ERROR_FAILURE): file /builds/worker/checkouts/gecko/dom/base/nsContentUtils.cpp:4333
[task 2023-03-06T00:49:59.881Z] 00:49:59     INFO - GECKO(6480) | [Parent 5064, Main Thread] WARNING: '!inner', file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorProtocol.cpp:163
[task 2023-03-06T00:49:59.881Z] 00:49:59     INFO - GECKO(6480) | [Parent 5064, Main Thread] WARNING: '!inner', file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorProtocol.cpp:163
[task 2023-03-06T00:49:59.896Z] 00:49:59     INFO - GECKO(6480) | Manager window unload handler
[task 2023-03-06T00:49:59.898Z] 00:49:59     INFO - GECKO(6480) | [Parent 5064, Main Thread] WARNING: '!wgc', file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorProtocol.cpp:168
[task 2023-03-06T00:49:59.907Z] 00:49:59     INFO - GECKO(6480) | [Child 4920: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 1497f487000 == 3 [pid = 4920] [id = 116]
[task 2023-03-06T00:49:59.908Z] 00:49:59     INFO - GECKO(6480) | [Child 4920: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 5 (1490414d470) [pid = 4920] [serial = 234] [outer = 0]
[task 2023-03-06T00:49:59.908Z] 00:49:59     INFO - GECKO(6480) | [Child 4920: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 6 (1497f489000) [pid = 4920] [serial = 235] [outer = 1490414d470]
[task 2023-03-06T00:49:59.911Z] 00:49:59     INFO - GECKO(6480) | Assertion failure: cx_->hadResourceExhaustion(), at /builds/worker/checkouts/gecko/js/src/jit/WarpOracle.cpp:192
[task 2023-03-06T00:50:00.065Z] 00:50:00     INFO -  Initializing stack-fixing for the first stack frame, this may take a while...
[task 2023-03-06T00:50:20.690Z] 00:50:20     INFO - GECKO(6480) | #01: js::jit::WarpOracle::createSnapshot() [js/src/jit/WarpOracle.cpp:191]
[task 2023-03-06T00:50:20.705Z] 00:50:20     INFO - GECKO(6480) | #02: js::jit::CreateWarpSnapshot(JSContext*, js::jit::MIRGenerator*, JS::Handle<JSScript *>) [js/src/jit/Ion.cpp:1557]
[task 2023-03-06T00:50:20.706Z] 00:50:20     INFO - GECKO(6480) | #03: js::jit::Compile(JSContext*, JS::Handle<JSScript *>, js::jit::BaselineFrame*, unsigned char*) [js/src/jit/Ion.cpp:1830]
[task 2023-03-06T00:50:20.706Z] 00:50:20     INFO - GECKO(6480) | #04: IonCompileScriptForBaseline(JSContext*, js::jit::BaselineFrame*, unsigned char*) [js/src/jit/Ion.cpp:2086]
[task 2023-03-06T00:50:20.707Z] 00:50:20     INFO - GECKO(6480) | #05: ??? (???:???)
[task 2023-03-06T00:50:20.707Z] 00:50:20     INFO - GECKO(6480) | [Child 9200, IPC I/O Child] WARNING: [43A0FEBD5386684F.66ECC2AD93DF071E]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-03-06T00:50:20.708Z] 00:50:20     INFO - GECKO(6480) | [Child 9200, IPC I/O Child] WARNING: [43A0FEBD5386684F.66ECC2AD93DF071E]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-03-06T00:50:20.708Z] 00:50:20     INFO - GECKO(6480) | Exiting due to channel error.
[task 2023-03-06T00:50:20.709Z] 00:50:20     INFO - GECKO(6480) | [GPU 8732, IPC I/O Child] WARNING: [8230320E13D8478.AB433625DF6543B5]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-03-06T00:50:20.709Z] 00:50:20     INFO - GECKO(6480) | [GPU 8732, IPC I/O Child] WARNING: [8230320E13D8478.AB433625DF6543B5]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-03-06T00:50:20.710Z] 00:50:20     INFO - GECKO(6480) | [GPU 8732, IPC I/O Child] WARNING: [8230320E13D8478.AB433625DF6543B5]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-03-06T00:50:20.710Z] 00:50:20     INFO - GECKO(6480) | [GPU 8732, Main Thread] WARNING: Shutting down GPU process early due to a crash!: file /builds/worker/checkouts/gecko/gfx/ipc/GPUParent.cpp:662
[task 2023-03-06T00:50:20.711Z] 00:50:20     INFO - GECKO(6480) | [GPU 8732, IPC I/O Child] WARNING: [8230320E13D8478.AB433625DF6543B5]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-03-06T00:50:20.712Z] 00:50:20     INFO - GECKO(6480) | [GPU 8732, IPC I/O Child] WARNING: [8230320E13D8478.AB433625DF6543B5]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-03-06T00:50:20.712Z] 00:50:20     INFO - GECKO(6480) | [GPU 8732, IPC I/O Child] WARNING: [8230320E13D8478.AB433625DF6543B5]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-03-06T00:50:20.713Z] 00:50:20     INFO - TEST-INFO | Main app process: exit 1
[task 2023-03-06T00:50:20.713Z] 00:50:20    ERROR - TEST-UNEXPECTED-FAIL | ShutdownLeaks | process() called before end of test suite
[task 2023-03-06T00:50:20.713Z] 00:50:20     INFO - TEST-INFO | Confirming we saw 496 DOCSHELL created and 471 destroyed log strings.
[task 2023-03-06T00:50:20.714Z] 00:50:20     INFO - TEST-INFO | Confirming we saw 1246 DOMWINDOW created and 1044 destroyed log strings.
[task 2023-03-06T00:50:20.714Z] 00:50:20     INFO - Buffered messages logged at 00:49:57
[task 2023-03-06T00:50:20.715Z] 00:50:20     INFO - Entering setup bound 
[task 2023-03-06T00:50:20.715Z] 00:50:20     INFO - Leaving setup bound 
[task 2023-03-06T00:50:20.715Z] 00:50:20     INFO - Entering test bound testExtensionList
[task 2023-03-06T00:50:20.716Z] 00:50:20     INFO - Loading manager window in tab
[task 2023-03-06T00:50:20.716Z] 00:50:20     INFO - Buffered messages logged at 00:49:59
[task 2023-03-06T00:50:20.717Z] 00:50:20     INFO - TEST-PASS | toolkit/mozapps/extensions/test/browser/browser_html_list_view_recommendations.js | Should have an add-ons manager window -
Flags: needinfo?(wmedina)
Whiteboard: [retriggered] → [retriggered][stockwell needswork:owner]

Looking into this. If I can catch it in rr locally I should be able to fix it.

Flags: needinfo?(wmedina)

Note: I've run this for hours locally without managing to catch anything.

Bug 1834038 fixes a bailout loop issue; crossing my fingers that it's the same one we're hitting here.

There have been 30 total failures in the last 7 days.
There are:

  • 16 failures on OS X 10.15 WebRender debug
  • 7 failures on Linux 18.04 x64 WebRender debug
  • 7 failures on Windows 11 x64 22H2 WebRender debug

Recent failure log.

[task 2023-07-16T04:04:50.883Z] 04:04:50     INFO - TEST-OK | browser/components/extensions/test/browser/browser_unified_extensions_accessibility.js | took 2061ms
[task 2023-07-16T04:04:50.892Z] 04:04:50     INFO - GECKO(7332) | [Child 2756: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 2ba88d3b800 == 2 [pid = 2756] [id = 213]
[task 2023-07-16T04:04:50.893Z] 04:04:50     INFO - GECKO(7332) | [Child 2756: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 3 (2ba8a7e47a0) [pid = 2756] [serial = 478] [outer = 0]
[task 2023-07-16T04:04:50.893Z] 04:04:50     INFO - GECKO(7332) | [Child 2756: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 4 (2ba88d3fc00) [pid = 2756] [serial = 479] [outer = 2ba8a7e47a0]
[task 2023-07-16T04:04:50.907Z] 04:04:50     INFO - checking window state
[task 2023-07-16T04:04:50.954Z] 04:04:50     INFO - TEST-START | browser/components/extensions/test/browser/browser_unified_extensions_cui.js
[task 2023-07-16T04:04:51.048Z] 04:04:51     INFO - GECKO(7332) | [GPU 3644, Compositor] WARNING: bad Shmem: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:450
[task 2023-07-16T04:04:51.057Z] 04:04:51     INFO - GECKO(7332) | [GPU 3644, Compositor] WARNING: bad Shmem: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:450
[task 2023-07-16T04:04:51.080Z] 04:04:51     INFO - GECKO(7332) | [Child 3620: Main Thread]: I/DocShellAndDOMWindowLeak --DOCSHELL 1fbe3544800 == 0 [pid = 3620] [id = 0] [url = https://example.org/]
[task 2023-07-16T04:04:51.127Z] 04:04:51     INFO - GECKO(7332) | [Child 3620: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 2 (1fbe07872e0) [pid = 3620] [serial = 1] [outer = 0] [url = https://example.org/]
[task 2023-07-16T04:04:51.133Z] 04:04:51     INFO - GECKO(7332) | [Child 3620: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 1 (1fbe3550400) [pid = 3620] [serial = 3] [outer = 0] [url = https://example.org/]
[task 2023-07-16T04:04:51.135Z] 04:04:51     INFO - GECKO(7332) | [Child 3620: Main Thread]: I/DocShellAndDOMWindowLeak --DOMWINDOW == 0 (1fbe354d000) [pid = 3620] [serial = 2] [outer = 0] [url = about:blank]
[task 2023-07-16T04:04:51.441Z] 04:04:51     INFO - GECKO(7332) | [Child 6368: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 243ab656400 == 6 [pid = 6368] [id = 1120]
[task 2023-07-16T04:04:51.455Z] 04:04:51     INFO - GECKO(7332) | [Child 6368: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 11 (243a6a8b020) [pid = 6368] [serial = 3217] [outer = 0]
[task 2023-07-16T04:04:51.456Z] 04:04:51     INFO - GECKO(7332) | [Child 6368: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 12 (243ab658000) [pid = 6368] [serial = 3218] [outer = 243a6a8b020]
[task 2023-07-16T04:04:51.504Z] 04:04:51     INFO - GECKO(7332) | [Child 6368: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 13 (243ad1cc400) [pid = 6368] [serial = 3219] [outer = 243a6a8b020]
[task 2023-07-16T04:04:51.516Z] 04:04:51     INFO - GECKO(7332) | [Child 6368, Main Thread] WARNING: NS_ENSURE_TRUE(mRequest) failed: file /builds/worker/checkouts/gecko/netwerk/base/nsBaseChannel.cpp:901
[task 2023-07-16T04:04:51.516Z] 04:04:51     INFO - GECKO(7332) | [Child 6368, Main Thread] WARNING: Failed to retarget HTML data delivery to the parser thread.: file /builds/worker/checkouts/gecko/parser/html/nsHtml5StreamParser.cpp:1235
[task 2023-07-16T04:04:51.551Z] 04:04:51     INFO - GECKO(7332) | [Child 6368: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 243ad1d1c00 == 7 [pid = 6368] [id = 1121]
[task 2023-07-16T04:04:51.556Z] 04:04:51     INFO - GECKO(7332) | [Child 6368: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 14 (243ad1917a0) [pid = 6368] [serial = 3220] [outer = 0]
[task 2023-07-16T04:04:51.557Z] 04:04:51     INFO - GECKO(7332) | [Child 6368: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 15 (243ae82cc00) [pid = 6368] [serial = 3221] [outer = 243ad1917a0]
[task 2023-07-16T04:04:51.557Z] 04:04:51     INFO - GECKO(7332) | [WARN  webrender::renderer::init] asking to enable_gpu_markers but no supporting extension was found
[task 2023-07-16T04:04:51.558Z] 04:04:51     INFO - GECKO(7332) | [Parent 5836, Main Thread] WARNING: Could not get window class style: file /builds/worker/checkouts/gecko/widget/windows/nsWindow.cpp:7554
[task 2023-07-16T04:04:51.614Z] 04:04:51     INFO - GECKO(7332) | [Parent 5836, Main Thread] WARNING: WebProgress Ignored: BrowsingContext is null or discarded: file /builds/worker/checkouts/gecko/dom/ipc/BrowserParent.cpp:3084
[task 2023-07-16T04:04:51.629Z] 04:04:51     INFO - GECKO(7332) | [Child 6368, Main Thread] WARNING: JSWindowActorChild::SendRawMessage (Conduits, ConduitClosed) not sent: !CanSend() || !mManager || !mManager->CanSend(): file /builds/worker/checkouts/gecko/dom/ipc/jsactor/JSWindowActorChild.cpp:61
[task 2023-07-16T04:04:51.640Z] 04:04:51     INFO - GECKO(7332) | [Child 1840: Main Thread]: I/DocShellAndDOMWindowLeak ++DOCSHELL 2cccd006800 == 3 [pid = 1840] [id = 232]
[task 2023-07-16T04:04:51.641Z] 04:04:51     INFO - GECKO(7332) | [Child 1840: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 6 (2ccca9e03e0) [pid = 1840] [serial = 523] [outer = 0]
[task 2023-07-16T04:04:51.641Z] 04:04:51     INFO - GECKO(7332) | [Child 1840: Main Thread]: I/DocShellAndDOMWindowLeak ++DOMWINDOW == 7 (2cccd007000) [pid = 1840] [serial = 524] [outer = 2ccca9e03e0]
[task 2023-07-16T04:04:51.642Z] 04:04:51     INFO - GECKO(7332) | [Parent 5836, Main Thread] WARNING: NS_ENSURE_TRUE(weakFrame.IsAlive()) failed: file /builds/worker/checkouts/gecko/layout/xul/nsXULPopupManager.cpp:1385
[task 2023-07-16T04:04:51.704Z] 04:04:51     INFO - GECKO(7332) | [Child 1840, Main Thread] WARNING: '!window', file /builds/worker/checkouts/gecko/dom/ipc/ContentChild.cpp:4250
[task 2023-07-16T04:04:51.712Z] 04:04:51     INFO - GECKO(7332) | [GPU 3644, Compositor] WARNING: Possibly dropping task posted to updater thread: file /builds/worker/checkouts/gecko/gfx/layers/apz/src/APZUpdater.cpp:371
[task 2023-07-16T04:04:51.724Z] 04:04:51     INFO - GECKO(7332) | [Parent 5836, Main Thread] WARNING: NS_ENSURE_TRUE(mNameHashtable.Get(aName, &index)) failed: file /builds/worker/checkouts/gecko/storage/mozStorageRow.cpp:91
[task 2023-07-16T04:04:51.736Z] 04:04:51     INFO - GECKO(7332) | [Parent 5836, Main Thread] WARNING: WebProgress Ignored: BrowsingContext is null or discarded: file /builds/worker/checkouts/gecko/dom/ipc/BrowserParent.cpp:3084
[task 2023-07-16T04:04:51.864Z] 04:04:51     INFO - GECKO(7332) | Assertion failure: cx_->hadResourceExhaustion(), at /builds/worker/checkouts/gecko/js/src/jit/WarpOracle.cpp:199
[task 2023-07-16T04:04:52.037Z] 04:04:52     INFO - GECKO(7332) | #01: js::jit::WarpOracle::createSnapshot() [js/src/jit/WarpOracle.cpp:198]
[task 2023-07-16T04:04:52.050Z] 04:04:52     INFO - GECKO(7332) | #02: js::jit::CreateWarpSnapshot(JSContext*, js::jit::MIRGenerator*, JS::Handle<JSScript *>) [js/src/jit/Ion.cpp:1602]
[task 2023-07-16T04:04:52.051Z] 04:04:52     INFO - GECKO(7332) | #03: js::jit::Compile(JSContext*, JS::Handle<JSScript *>, js::jit::BaselineFrame*, unsigned char*) [js/src/jit/Ion.cpp:1875]
[task 2023-07-16T04:04:52.052Z] 04:04:52     INFO - GECKO(7332) | #04: IonCompileScriptForBaseline(JSContext*, js::jit::BaselineFrame*, unsigned char*) [js/src/jit/Ion.cpp:2131]
[task 2023-07-16T04:04:52.052Z] 04:04:52     INFO - GECKO(7332) | #05: ??? (???:???)
[task 2023-07-16T04:04:52.319Z] 04:04:52     INFO - GECKO(7332) | [GPU 3644, IPC I/O Child] WARNING: [636540DADE8570C.C6D46F16AAB1B621]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-07-16T04:04:52.319Z] 04:04:52     INFO - GECKO(7332) | [Child 5780, IPC I/O Child] WARNING: [30D9E5D1E4E097F9.B13241748AAA5621]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-07-16T04:04:52.319Z] 04:04:52     INFO - GECKO(7332) | [Child 2564, IPC I/O Child] WARNING: [A15806773BA40A14.B68B2CD2D4F9A268]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-07-16T04:04:52.319Z] 04:04:52     INFO - GECKO(7332) | [Child 2564, IPC I/O Child] WARNING: [A15806773BA40A14.B68B2CD2D4F9A268]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-07-16T04:04:52.319Z] 04:04:52     INFO - GECKO(7332) | Exiting due to channel error.
[task 2023-07-16T04:04:52.319Z] 04:04:52     INFO - GECKO(7332) | [Child 8040, IPC I/O Child] WARNING: [41868F3399EC1176.636B9A70B7B20E06]: Ignoring message 'EVENT_MESSAGE' to peer 1.1 due to a missing broker: file /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:352
[task 2023-07-16T04:04:52.319Z] 04:04:52     INFO - GECKO(7332) | Exiting due to channel error.
[task 2023-07-16T04:04:52.460Z] 04:04:52     INFO - TEST-INFO | Main app process: exit 1

Hello Will! Could you please take another look at this bug? Is there any chance you can assign this to someone?

Thank you!

Flags: needinfo?(wmedina)
Whiteboard: [retriggered][stockwell unknown] → [retriggered][stockwell needswork:owner]

This is very hard to debug without being able to reproduce locally. A few months ago I spent a couple days running some of these tests in rr in a loop on my build machine and didn't capture anything. Not sure if this spike is random noise or if something has changed; if the numbers are still high next week, I'll give it another go locally.

Assignee: nobody → iireland
Flags: needinfo?(wmedina)

A pernosco recording would also be enough to make forward progress, if we could get one somehow.

Update

There have been 39 total failures within the last 7 days:

  • 5 failures on Linux 18.04 x64 WebRender debug
  • 22 failures on OS X 10.15 WebRender debug
  • 5 failures on Windows 11 x86 22H2 WebRender debug
  • 7 failures on Windows 11 x64 22H2 WebRender debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=424088254&repo=autoland&lineNumber=5693

Update

There have been 62 total failures within the last 7 days:

  • 12 failures on Linux 18.04 x64 WebRender debug
  • 26 failures on OS X 10.15 WebRender debug
  • 10 failures on Windows 11 x86 22H2 WebRender debug
  • 14 failures on Windows 11 x64 22H2 WebRender debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=425046579&repo=autoland&lineNumber=23795

[task 2023-08-05T12:20:59.620Z] 12:20:59     INFO - GECKO(2246) | Assertion failure: cx_->hadResourceExhaustion(), at /builds/worker/checkouts/gecko/js/src/jit/WarpOracle.cpp:199
[task 2023-08-05T12:20:59.621Z] 12:20:59     INFO - GECKO(2246) | #01: js::jit::WarpOracle::createSnapshot() [js/src/jit/WarpOracle.cpp:198]
[task 2023-08-05T12:20:59.622Z] 12:20:59     INFO - GECKO(2246) | #02: js::jit::CreateWarpSnapshot(JSContext*, js::jit::MIRGenerator*, JS::Handle<JSScript*>) [js/src/jit/Ion.cpp:1605]
[task 2023-08-05T12:20:59.623Z] 12:20:59     INFO - GECKO(2246) | #03: js::jit::Compile(JSContext*, JS::Handle<JSScript*>, js::jit::BaselineFrame*, unsigned char*) [js/src/jit/Ion.cpp:1866]
[task 2023-08-05T12:20:59.624Z] 12:20:59     INFO - GECKO(2246) | #04: IonCompileScriptForBaseline(JSContext*, js::jit::BaselineFrame*, unsigned char*) [js/src/jit/Ion.cpp:2122]

This has reached our disable-recommended queue.
Will, do you know who can help us with a pernosco recording?
Thank you.

Flags: needinfo?(wmedina)
Flags: needinfo?(wmedina) → needinfo?(iireland)

Matt, this is the bug we talked about briefly at All-Hands, where we thought your beefier desktop might be able to help.

As you can see, the failures are scattered around pretty much at random. It doesn't seems specific to any particular test. It does seem to occur relatively more often on Mac. I'm not sure what to take away from that.

My least bad suggestion is to pick a mochitest that is known to trigger this and run it in a loop with rr chaos mode for a while to see if anything turns up.

Flags: needinfo?(iireland) → needinfo?(mgaudet)

I encountered this assert when fuzzing on git commit c3b462663f7d45d2d1fc91374a68856b484a9a29. The sample is a bit flaky, but almost half of the executions trigger the assert. Its rather sensitive to the gc settings, so you might need to fiddle with the second parameter to gc-zeal.
I can also upload a pernosco session if you'd like.

obj-x86_64-pc-linux-gnu/dist/bin/js --baseline-warmup-threshold=10 --ion-warmup-threshold=100 --ion-check-range-analysis --ion-extra-checks --fuzzing-safe --disable-oom-functions --enable-array-grouping --enable-new-set-methods --enable-well-formed-unicode-strings --small-function-length=2048 --inlining-entry-threshold=32 --gc-zeal=23,129 crash.js

function main() {
    class C2 {
        c = "saPW";
        #toString(a4, a5) {
        }
        3 = "iso8601";
    }   
    const v6 = new C2();
    function f9() {
        class C10 extends C2 {
            static 9 = v6; 
            c;
            2 = 15n;
        }
        new C10();
        new C10();
        const v13 = new C10();
        v13[20] = -5n;
        const t18 = "saPW";
        t18[8] = -5n;
        const o14 = { 
        };
    }   
    for (let v15 = 0; v15 < 100; v15++) {
        f9();
    }   
}
for (let v16 = 0; v16 < 4; v16++) {
    main();
}

I don't know if this is the specific issue we've been running into in this bug, but it's definitely a good catch.

This looks like an awkward interaction between bailout loop detection, stub folding, and Jon's patches to discard IC stubs with pointers to dead shapes.

AFAICT, most of the testcase is only necessary to get the GC timing to line up. Simplifying, the core of the bug looks like this:

function foo() {
        class C {
            x = 1; // Initializer
        }
        new C();
}
for (let i = 0; i < N; i++) {
  foo();
}

We have a class defined inside a function. The constructor for that class calls an initializer function to set the value of x. Inside the initializer, we have an InitProp IC. Because we keep defining new versions of the class, the shape of the object that we see in that IC keeps changing.

The first six shapes create separate ICs. While attaching the seventh, we trigger stub folding, creating a single IC with a GuardMultipleShapes op. At some point we transpile this to Ion. We keep adding new shapes to the GuardMultipleShapes until it hits MaxFoldedShapes (16).

New shapes keep being generated. We bail out of Ion; because the GuardMultipleShapes is full, we attach new stubs for those shapes. Eventually, we've bailed out too often, and we invalidate. Up to this point, everything is working as intended. The IC chain has new stubs in it, so we won't recompile with the same data.

Then we trigger a GC. All the shapes we've generated are dead. We remove the useless stubs. But now the only stub left is the GuardMultipleShapes stub. When we recompile, our IC hash is the same: the address of the folded stub hasn't changed, and it's once again the only stub.

Here's a testcase that hits the same problem more reliably:

// |jit-test| --fast-warmup; --no-threads; --blinterp-eager

function foo(o) {
  return o.x;
}

function bar(n) {
  with ({}) {}
  class C {}
  for (var i = 0; i < n; i++) {
    var c = new C();
    c.x = 0;
    foo(c);
  }
}

with ({}) {}

function triggerIonCompile() {
  for (var i = 0; i < 10; i++) {
    bar(3);
  }
}

triggerIonCompile();

// Fill up shape list
for (var i = 0; i < 6; i++) {
  bar(10);
}

// Overflow shape list, adding a new baseline IC.
bar(10);

// Discard that IC.
gc();

triggerIonCompile();

I'll figure out a fix next week.

So it seems like it's not so much a test as it is a suite that triggers this; tried running a suite under rr chaos overnight with a debug build and more than twelve hours later hadn't finished the suite (and the rr recordings were already 8GB).

Here was my planed attack script

mkdir recordings
for i in `seq 0 1000`;
do 
        mkdir recordings/$i
        export _RR_TRACE_DIR=$PWD/recordings/$i
        (./mach mochitest --use-http3-server --flavor=browser --subsuite=devtools --chunk-by-runtime --timeout=1200 --headless --debugger pernosco-record --debugger-args ' --chaos' && rm -rf $PWD/recordings/$i) || break;
done;

(Honestly, needs parallelism but I wanted to make sure this worked first) This was roughed out of a linux failure here.

This morning though, I looked through the linux failures and saw two failing the same test, so I'm going to try running just that one test (browser_inspector_keyboard-shortcuts-copy-outerhtml.js, and it runs much faster than the whole devtools suite!) in a loop. We'll see how this goes.

Flags: needinfo?(mgaudet)
Depends on: 1851911

I've split the testcase out into bug 1851911, leaving this bug for the intermittent failures. These failures started before Jon landed weak shape pointers in IC stubs, although the timeline of that patch (bug 1837620, landed July 7) does line up with the uptick in CI failures from single to double digits.

So far, I've been unable to capture this.

I keep getting unrelated leak failures (with un-replayable recordings unfortunately, and so I've yet to open a different bug)

Parallel execution of mochitest hasn't panned out -- the various servers collide unfortunately.

Going to try a weekend run; one thread but without trying to bail on failure. We'll see what happens. Unfortunately repro is relatively slow.

So at about 2000 recordings, I've still not reproduced this.

I'm going to suggest that the next step here would be to alter the assertion to dump the relevant information (CacheIR spew etc) before crashing and see if we can get automation to dump something interesting.

I've landed two fixes in the last week that could plausibly bring the crash rate down here (bug 1851911 and bug 1852398). Let's see what the numbers look like next week.

Dumping useful information is hard. We have the CacheIR available, but we don't know why we bailed out before, or why we didn't update anything.

Interesting that the only failures yesterday were in beta. That might be a good sign, or it might just mean that somebody was running a lot of beta tests.

Iain, please note that the /webdriver/tests/classic/perform_actions/key_events.py | test_special_key_sends_keydown seems to trigger this specific failure quite a lot. 28 out of 37 failures from last week have been triggered by that particular test. With the Pytest parameterization there is still some variation, but there should be a probably good chance to get it reproduced with a debug build on Windows.

There is also bug 1852893 with a(nother) deterministic reproducer of this assert.

Oh, bug 1852893 somehow fell off my radar. I'll take a look at that today.

Nearly all of the crashes last week happened on beta, which does not have the fixes for bug 1851911 and bug 1852398. There was only one nightly failure, which seems very promising. We could request beta uplift, but we're shipping in about a week anyway, so maybe it's best to just let this ride the trains. If we are still seeing a lot of failures on that test, we can look into it.

After cherry-picking the fix in bug 1852893 fuzzing no longer triggers this assert.

Iain, shall we close this bug given that no other failure has been seen?

Flags: needinfo?(iireland)

When this bug was first opened we were seeing a couple failures every week or so. It looks like we've fixed the big spike, but it might still be too soon to say that we've caught everything.

Flags: needinfo?(iireland)

Last failure here was on 20th Sep https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2023-09-13&endday=2023-10-13&tree=trunk&failurehash=all&bug=1803419, I think we could say it is fixed. We'll open a new one if it's the case.

Status: NEW → RESOLVED
Closed: 1 year ago
status-firefox119: affected → fixed
status-firefox120: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 120 Branch
Whiteboard: [retriggered][stockwell disable-recommended] → [retriggered]
You need to log in before you can comment on or make changes to this bug.