Assertion failure: mCount == 0 (AddPurpleRoot failed), at /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp:1065
Categories
(Core :: Cycle Collector, defect)
Tracking
()
| Tracking | Status | |
|---|---|---|
| firefox-esr91 | --- | unaffected |
| firefox-esr102 | --- | unaffected |
| firefox104 | --- | unaffected |
| firefox105 | --- | unaffected |
| firefox106 | --- | wontfix |
| firefox107 | --- | wontfix |
| firefox108 | --- | wontfix |
People
(Reporter: tsmith, Unassigned)
References
(Blocks 1 open bug)
Details
(6 keywords)
Attachments
(7 files)
This (or a variation of this) was first found while fuzzing m-c 20220818-6502583dede7 (--enable-debug --enable-fuzzing)
Assertion failure: mCount == 0 (AddPurpleRoot failed), at /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp:1065
eax = 0xb95d9ad4 ebp = 0xf2efe298 ebx = 0xbd112a24
ecx = 0xbdd1dc74 edi = 0xab0c50dc edx = 0x00000000
eflags = 0x00210286 eip = 0xb2ca1b74 esi = 0x00000429
esp = 0xf2efe270
OS|Android|0.0.0 Linux 5.4.86-android11-2-00006-gae78026f427c-ab7595864 #1 SMP PREEMPT Thu Jul 29 20:54:47 UTC 2021 i686
CPU|x86|GenuineIntel family 6 model 6 stepping 3|4
Crash|SIGSEGV / SEGV_MAPERR|0x00000000|48
48|0|libxul.so|nsPurpleBuffer::SelectPointers(CCGraphBuilder&)|hg:hg.mozilla.org/mozilla-central:xpcom/base/nsCycleCollector.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|1065|0x7a
48|1|libxul.so|nsCycleCollector::BeginCollection(mozilla::CCReason, ccIsManual, nsICycleCollectorListener*)|hg:hg.mozilla.org/mozilla-central:xpcom/base/nsCycleCollector.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|3620|0x2e1
48|2|libxul.so|nsCycleCollector::Collect(mozilla::CCReason, ccIsManual, js::SliceBudget&, nsICycleCollectorListener*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/base/nsCycleCollector.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|3412|0x11a
48|3|libxul.so|nsCycleCollector_collect(mozilla::CCReason, nsICycleCollectorListener*)|hg:hg.mozilla.org/mozilla-central:xpcom/base/nsCycleCollector.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|3920|0x87
48|4|libxul.so|mozilla::dom::workerinternals::(anonymous namespace)::WorkerJSRuntime::CustomGCCallback(JSGCStatus)|hg:hg.mozilla.org/mozilla-central:dom/workers/RuntimeService.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|816|0x42
48|5|libxul.so|mozilla::CycleCollectedJSRuntime::OnGC(JSContext*, JSGCStatus, JS::GCReason)|hg:hg.mozilla.org/mozilla-central:xpcom/base/CycleCollectedJSRuntime.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|1889|0xec
48|6|libxul.so|mozilla::CycleCollectedJSRuntime::GCCallback(JSContext*, JSGCStatus, JS::GCReason, void*)|hg:hg.mozilla.org/mozilla-central:xpcom/base/CycleCollectedJSRuntime.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|1035|0x56
48|7|libxul.so|js::gc::GCRuntime::callGCCallback(JSGCStatus, JS::GCReason) const|hg:hg.mozilla.org/mozilla-central:js/src/gc/GC.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|1301|0x6c
48|8|libxul.so|js::gc::GCRuntime::maybeCallGCCallback(JSGCStatus, JS::GCReason)|hg:hg.mozilla.org/mozilla-central:js/src/gc/GC.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|3735|0xd1
48|9|libxul.so|js::gc::GCRuntime::gcCycle(bool, js::SliceBudget const&, JS::GCReason)|hg:hg.mozilla.org/mozilla-central:js/src/gc/GC.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|3825|0x923
48|10|libxul.so|js::gc::GCRuntime::collect(bool, js::SliceBudget const&, JS::GCReason)|hg:hg.mozilla.org/mozilla-central:js/src/gc/GC.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|4007|0x219
48|11|libxul.so|js::gc::GCRuntime::gc(JS::GCOptions, JS::GCReason)|hg:hg.mozilla.org/mozilla-central:js/src/gc/GC.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|4084|0x7f
48|12|libxul.so|JS::NonIncrementalGC(JSContext*, JS::GCOptions, JS::GCReason)|hg:hg.mozilla.org/mozilla-central:js/src/gc/GCAPI.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|289|0x5b
48|13|libxul.so|mozilla::dom::WorkerPrivate::GarbageCollectInternal(JSContext*, bool, bool)|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerPrivate.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|5218|0xa8
48|14|libxul.so|mozilla::dom::(anonymous namespace)::GarbageCollectRunnable::WorkerRun(JSContext*, mozilla::dom::WorkerPrivate*)|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerPrivate.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|801|0x2a
48|15|libxul.so|mozilla::dom::WorkerRunnable::Run()|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerRunnable.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|377|0x48f
48|16|libxul.so|mozilla::dom::WorkerPrivate::ProcessAllControlRunnablesLocked()|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerPrivate.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|3782|0x8b
48|17|libxul.so|mozilla::dom::WorkerPrivate::RunCurrentSyncLoop()|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerPrivate.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|4171|0x165
48|18|libxul.so|mozilla::dom::AutoSyncLoopHolder::Run()|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerPrivate.h:95412593f3f1a381b55ce360fdef6c59debec24c|1504|0x6b
48|19|libxul.so|mozilla::dom::workerinternals::(anonymous namespace)::LoadAllScripts(mozilla::dom::WorkerPrivate*, mozilla::UniquePtr<mozilla::dom::SerializedStackHolder, mozilla::DefaultDelete<mozilla::dom::SerializedStackHolder> >, nsTArray<nsTString<char16_t> > const&, bool, mozilla::dom::WorkerScriptType, mozilla::ErrorResult&, mozilla::Encoding const*)|hg:hg.mozilla.org/mozilla-central:dom/workers/ScriptLoader.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|273|0x1c8
48|20|libxul.so|mozilla::dom::workerinternals::LoadMainScript(mozilla::dom::WorkerPrivate*, mozilla::UniquePtr<mozilla::dom::SerializedStackHolder, mozilla::DefaultDelete<mozilla::dom::SerializedStackHolder> >, nsTSubstring<char16_t> const&, mozilla::dom::WorkerScriptType, mozilla::ErrorResult&, mozilla::Encoding const*)|hg:hg.mozilla.org/mozilla-central:dom/workers/ScriptLoader.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|1212|0x63
48|21|libxul.so|mozilla::dom::(anonymous namespace)::CompileScriptRunnable::WorkerRun(JSContext*, mozilla::dom::WorkerPrivate*)|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerPrivate.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|380|0xd8
48|22|libxul.so|mozilla::dom::WorkerRunnable::Run()|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerRunnable.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|377|0x48f
48|23|libxul.so|nsThread::ProcessNextEvent(bool, bool*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|1199|0x6a7
48|24|libxul.so|NS_ProcessNextEvent(nsIThread*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|465|0x6c
48|25|libxul.so|mozilla::dom::WorkerPrivate::DoRunLoop(JSContext*)|hg:hg.mozilla.org/mozilla-central:dom/workers/WorkerPrivate.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|3204|0x305
48|26|libxul.so|mozilla::dom::workerinternals::(anonymous namespace)::WorkerThreadPrimaryRunnable::Run()|hg:hg.mozilla.org/mozilla-central:dom/workers/RuntimeService.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|2042|0x3f9
48|27|libxul.so|nsThread::ProcessNextEvent(bool, bool*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|1199|0x6a7
48|28|libxul.so|NS_ProcessNextEvent(nsIThread*, bool)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThreadUtils.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|465|0x6c
48|29|libxul.so|mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)|hg:hg.mozilla.org/mozilla-central:ipc/glue/MessagePump.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|300|0xb5
48|30|libxul.so|MessageLoop::RunInternal()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:95412593f3f1a381b55ce360fdef6c59debec24c|381|0x66
48|31|libxul.so|MessageLoop::Run()|hg:hg.mozilla.org/mozilla-central:ipc/chromium/src/base/message_loop.cc:95412593f3f1a381b55ce360fdef6c59debec24c|356|0x3c
48|32|libxul.so|nsThread::ThreadFunc(void*)|hg:hg.mozilla.org/mozilla-central:xpcom/threads/nsThread.cpp:95412593f3f1a381b55ce360fdef6c59debec24c|384|0x13d
48|33|libnss3.so|_pt_root|hg:hg.mozilla.org/mozilla-central:nsprpub/pr/src/pthreads/ptthread.c:95412593f3f1a381b55ce360fdef6c59debec24c|201|0x14b
48|34|libc.so||||
48|35|libc.so||||
48|36|libc.so||||
48|37|libc.so||||
48|38|libnss3.so|pt_recvfrom_cont|hg:hg.mozilla.org/mozilla-central:nsprpub/pr/src/pthreads/ptio.c:95412593f3f1a381b55ce360fdef6c59debec24c|996|0x7a
|
Reporter | |
Comment 1•4 years ago
|
||
Recently a large number of issues that all have (I believe) a similar root case have been reported while running DOM fuzzing on Android (32-bit debug). The fact that the same test case can crash in different ways makes it very difficult to bucket and manage the results.
Some of the variations include:
Assertion failure: false (OOM while building cycle collector graph), at /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp:1997
Assertion failure: info.numArenasFree <= ArenasPerChunk, at /builds/worker/checkouts/gecko/js/src/gc/Heap.cpp:618
Assertion failure: thing->isPermanentAndMayBeShared(), at /builds/worker/checkouts/gecko/js/src/gc/Marking.cpp:151
Assertion failure: !IsInsideNursery(obj) (obj shouldn't live in nursery.), at /builds/worker/checkouts/gecko/js/src/gc/Tenuring.cpp:195
Assertion failure: isTenured(), at /builds/worker/checkouts/gecko/js/src/gc/Cell.h:303
Assertion failure: false (OOM while building cycle collector graph), at /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp:1997
All of them have mozilla::dom::WorkerPrivate::GarbageCollectInternal in the stack.
|
||
Updated•4 years ago
|
|
|
||
Comment 2•4 years ago
|
||
The original failure looks like a cycle collector issue, as do the "OOM while building cycle collector graph" assertions, but I'm not sure about the others. Do you have stacks for those?
All cycle collections on DOM Workers originate inside the GC.
|
|
Reporter | |
Comment 3•4 years ago
|
||
|
|
Reporter | |
Comment 4•4 years ago
|
||
|
|
Reporter | |
Comment 5•4 years ago
|
||
|
|
Reporter | |
Comment 6•4 years ago
|
||
|
|
Reporter | |
Comment 7•4 years ago
|
||
|
|
Reporter | |
Comment 8•4 years ago
|
||
|
|
||
Comment 9•4 years ago
|
||
I'm going to hide this for now. Such a wide ranging list of GC and CC crashes could indicate a bug in the memory management of one of the DOM implementations used in the test case.
|
||
Comment 10•4 years ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #2)
All cycle collections on DOM Workers originate inside the GC.
For assert_2.txt I cannot really see any worker involved.
The others but assert_6.txt are all inside a WorkerPrivate::RunCurrentSyncLoop, the latter is inside the final GCCC of WorkerThreadPrimaryRunnable::Run.
The test case is basically creating a bunch of workers that do not much (based on a blob's content) and posting a message to each of them.
|
|
||
Comment 11•4 years ago
•
|
||
This looks a lot like memory corruption, so I'm going to mark it sec-high. We can adjust the rating later if needed.
Tyson said there's no automation for bisection on Android, but that this is frequent enough that it was probably introduced on the build mentioned in comment 0 (mozilla-central 20220818-6502583dede7) or from something 6 hours earlier. I'll try looking through commit logs around then for things that look related. Bug 1771423 landed on 8-15, so a bit too early.
|
|
||
Comment 12•4 years ago
|
||
Tyson, does this reproduce without the
const a = new ImageData(7105, 7105)
line? This could create some memory pressure condition? Not sure about the underlying buffer, but I'd assume this to be at least ~150MB, which on a simulated android device might be a lot?
|
|
||
Comment 13•4 years ago
|
||
Yeah, given that this is only happening on 32-bit Android, it feels like memory pressure is going to be involved here. I dunno if adding some forceGC() calls here and there might help reproduce otherwise, or maybe it requires pressure when there's no JS on the stack.
|
|
Reporter | |
Comment 14•4 years ago
|
||
(In reply to Jens Stutte [:jstutte] from comment #12)
Tyson, does this reproduce without the
const a = new ImageData(7105, 7105)line?
The issue does not reproduce when that line is removed from the test case.
This could create some memory pressure condition? Not sure about the underlying buffer, but I'd assume this to be at least ~150MB, which on a simulated android device might be a lot?
FWIW I am running a 32-bit build on 64-bit Android 11 with 6GB of memory.
|
|
Reporter | |
Comment 15•4 years ago
|
||
I tested with m-c 20220818-7ed0939bcec0 (the build prior to 20220818-6502583dede7) and I was unable to reproduce the issue.
|
|
||
Comment 16•4 years ago
|
||
Olli, I do not know enough about the details here, but I can imagine that
- the memory pressure triggers a CC on the main thread while we are running the loop
- while the worker endings will trigger n CC on the worker thread(s) (not sure about the android thread model)
and probably at least the blob object is shared between all those threads. I'd assume that none of the objects can really be freed while we are looping (except the workers itself) and we are running some parallel in vane CC, probably in a low memory condition (in fact there is some OOM among the triggered cases).
How resilient is the CC against OOM? Is there some attempt to detect OOM but we might not recover well from it?
|
|
||
Comment 17•4 years ago
|
||
(In reply to Tyson Smith [:tsmith] from comment #15)
I tested with m-c 20220818-7ed0939bcec0 (the build prior to 20220818-6502583dede7) and I was unable to reproduce the issue.
It looks like there were two commits to m-c after 20220818-7ed0939bcec0, so one in addition to 20220818-6502583dede7.
The first backed out some patches. The second one backed out some other patches and also landed bug 1781046.
Emilio, any chance the test case in this bug could cause issues with the resize observer? It doesn't look like it to me but maybe I'm wrong.
|
||
Comment 18•4 years ago
|
||
No, I don't see how either... https://hg.mozilla.org/mozilla-central/rev/f83b42c4d287 touched CC code but it was fairy trivial and looks good to me (plus there should be no resizeobserver involved).
|
|
||
Comment 19•4 years ago
|
||
Strange. That patch looks okay to me, too.
"OOM while building cycle collector graph" is definitely a CC OOM. Looking at the code "AddPurpleRoot failed" is probably also the sign of an OOM in the CC, but during the initial purple buffer scanning. So it could be that these failures are just the sign of running out of memory, in which case this isn't really a security issue. I'm not sure about the GC issues.
|
||
Comment 20•4 years ago
|
||
MOZ_ASSERT(mCount == 0, "AddPurpleRoot failed"); looks like a bogus assertion. It might be triggered when we're OOMing, but I think it may happen also if we just don't find any roots, which in theory is possible.
The CC failures look OOM issues to me, but I'm not sure about the ones happening in GC.
So, basically I answered similarly to mccr8 :)
|
|
||
Comment 21•4 years ago
|
||
Jon, how dangerous looking are those GC asserts? Are they a sign of OOM like the CC asserts?
|
||
Comment 22•4 years ago
|
||
Individually they look pretty bad, but taken together it looks more like memory corruption. I don't think OOM could produce this.
|
||
Comment 23•4 years ago
|
||
I think this is a better component for this?
|
|
||
Comment 24•4 years ago
|
||
This is an OOM in the cycle collector. XPCOM is the cycle collector component. The GC is in the stack because on workers all CCs are triggered by the GC.
|
|
||
Comment 25•4 years ago
|
||
Well, I guess it is kind of all over the place. The actual issue is likely not in the GC or CC, IMO.
|
|
||
Comment 26•4 years ago
|
||
Maybe workers is a more sensible component. The test case is basically "spam allocation in workers".
|
|
||
Comment 27•4 years ago
|
||
So looking again at the testcase:
<script id="worker1" type="javascript/worker">
let x = 0
</script>
<script>
const a = new ImageData(7105, 7105)
window.addEventListener('load', async () => {
const blob = new Blob([document.querySelector('#worker1').textContent], { type: "text/javascript" })
for (let i = 0; i < 17; i++) {
let worker = new Worker(window.URL.createObjectURL(blob));
worker.postMessage([a], [])
}
})
</script>
I see actually that we create 17 worker and to each of them we postMessage([a], []), that is the big image. Do we copy that data 17 times here? Then this looks like a simple DOS to me and not very actionable.
|
|
||
Comment 28•4 years ago
|
||
The issue is that the GC failures kind of look like memory corruption (comment 22), so it seems like more than resource exhaustion. Likely something (the JS engine?) is failing to allocate something somewhere due to all of the memory and things are going awry, but it is hard to say what exactly.
|
||
Comment 29•4 years ago
•
|
||
(In reply to Jens Stutte [:jstutte] from comment #27)
I see actually that we create 17 worker and to each of them we
postMessage([a], []), that is the big image. Do we copy that data 17 times here? Then this looks like a simple DOS to me and not very actionable.
The ImageData allocation is 7105 * 7105 * 4 = ~192.6 MiB as a Uint8 clampd array. Each postMessage call will serialize the typed array, so yes, 17 copies of that data (in addition to the original copy). (No attempt is made to transfer in this test case.)
|
|
||
Comment 30•4 years ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #28)
The issue is that the GC failures kind of look like memory corruption (comment 22), so it seems like more than resource exhaustion. Likely something (the JS engine?) is failing to allocate something somewhere due to all of the memory and things are going awry, but it is hard to say what exactly.
OK, I think the stack traces alone will not really help us here. Maybe if we had pernosco traces for the different cases we could look if there was some failed fallible allocation before we fail (and where) ?
|
|
||
Comment 31•4 years ago
|
||
The problem is that this is on 32-bit Android, so we can't get Pernosco traces, as far as I know. I'm not sure if there's some way to constrain memory on Linux in a way that will induce the same kind of OOM issues.
|
|
||
Comment 32•4 years ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #31)
The problem is that this is on 32-bit Android, so we can't get Pernosco traces, as far as I know. I'm not sure if there's some way to constrain memory on Linux in a way that will induce the same kind of OOM issues.
Oh right, sorry for the noise. We might want to try what happens if we scale up the number of workers under normal linux?
|
|
||
Comment 33•4 years ago
|
||
I wonder if bug 1791975 could be related to this. The test case here does kind of look like "OOM while creating a global".
|
|
||
Updated•4 years ago
|
|
|
||
Comment 34•4 years ago
•
|
||
(In reply to Andrew McCreight [:mccr8] from comment #31)
The problem is that this is on 32-bit Android, so we can't get Pernosco traces, as far as I know. I'm not sure if there's some way to constrain memory on Linux in a way that will induce the same kind of OOM issues.
I understand that under linux we have the following options.
- The bash built-in
ulimit -Sv <bytes>which will impact the bash process and its children in terms of the max virtual memory size of any given process, I think. Therefore we'd want this invocation to happen inside whatever rr is wrapping so rr itself is not impacted and it seems like this can simulate the 32-bit address space limit well since it's not a group limit. - Calling
setrlimit(RLIMIT_AS, ...)which is what bash uses too and has units of bytes. - It turns out under DEBUG builds on linux we will call setrlimit ourselves with the value of the "MOZ_MEM_LIMIT" environment variable which appears to have a unit of megabytes. We might need to augment the ifdef in use here; I never remember if the fuzzing defines set DEBUG or if they just tend to enable the same things.
:twsmith is it possible to try and reproduce this under rr/pernosco using MOZ_MEM_LIMIT so that we can artificially limit the address space of the processes under test? It sounds like this need not involve android at all then, maybe?
|
|
||
Comment 35•4 years ago
|
||
The test case in bug 1789371 is also vaguely similar: it spams a bunch of workers that allocate a bunch of stuff in a 32-bit build and then we end up with UAFs.
|
||
Updated•4 years ago
|
|
|
||
Comment 36•4 years ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #35)
The test case in bug 1789371 is also vaguely similar: it spams a bunch of workers that allocate a bunch of stuff in a 32-bit build and then we end up with UAFs.
It would be really helpful if based on this or bug 1789371 (which I can't see) we were able to reproduce it under rr/pernosco. IIUC rr supports 32 Bit linux and pernosco hopefully too. :tsmith, some help from the fuzzer team here would be really appreciated! Thanks
|
|
||
Comment 37•4 years ago
|
||
(In reply to Andrew McCreight [:mccr8] from comment #35)
The test case in bug 1789371 is also vaguely similar: it spams a bunch of workers that allocate a bunch of stuff in a 32-bit build and then we end up with UAFs.
Now that I have access there: The patch talks about extensions, so most likely it will not help here, IIUC.
|
|
||
Comment 38•4 years ago
|
||
Yeah, this test case doesn't involve canvas so unfortunately it seems like something else. It at least gives an idea of what the bug in the code migth look like, somewhere.
|
|
Reporter | |
Comment 39•4 years ago
|
||
I found a test case that I can reproduce the issue with on Windows with a x86 debug build. Truber has docker setup for repro'ing with x86 Linux builds under rr.
It may takes some time to reproduce since the test case is not super reliable and rr is slow.
|
|
Reporter | |
Comment 40•4 years ago
|
||
So far no luck reproducing the issue with a no-opt debug build or with rr (debug or no-opt debug). We are able to reproduce the issue on Linux x64 with a 32-bit browser build, so that's progress.
|
|
Reporter | |
Comment 41•4 years ago
|
||
We have made some progress.
Running 32-bit builds under rr with --chaos is now fixed.
Some test cases have triggered similar assertion running with no-opt builds.
Pernosco does not and will not support 32-bit builds without (months?) of work on their side (this is not a problem until we get a useful trace anyways)
Running and re-running the test cases with rr + no-opt builds is very slow going. I will continue to do so in the lab and hopefully capture something helpful. Until then I'll mark this stalled. It is still a fuzzblocker because it is common and the assortment of assertions/crashes make it difficult to bucket.
|
|
||
Comment 42•4 years ago
|
||
For future discussion or anyone planning next steps, just want to echo some thoughts I expressed in discussion with :tsmith on matrix and that I think are consistent with discussions I've had with :jstutte via other channels:
- Any next steps for these GC-related assertions need not be worker-specific. Although the specific test case is good at making crashes happen on the worker thread (5 of the 6 crashes are on worker threads), the crashes are all in CC (1/6 asserts) or JS GC code (5/6 asserts). This makes sense because the test case creates 17 workers which will all trigger GCs either due to being idle and/or due to being terminated because nothing is done to keep the workers alive and so they will get GCed.
- In particular, WorkerThreadPrimaryRunnable is present and active on all 5/5 worker asserts. This means that we're not seeing any situations where weird UAFs could be happening with the worker lifecycle.
- That said, the bulk of the worker machinery happening other than the GCs is script loading, but this is a consistent / straightforward script load and any problems in script loading thus far have been directly implicated. It's also the case that the JS script loading is using straightforward JS APIs and there isn't particularly any potential for workers code to do weird things.
- It seems reasonable that the fuzzer could take steps to notice when a process is getting close to memory/address space exhaustion and maybe adjust its constants that potentially correlate to memory usage back downwards and/or give GCs time to clean up? (FuzzingFunctions could expose the VM size and allow the fuzzer to realize when it just did something that allocated a lot of memory.)
- Unless the intent is to test the OOM edge-cases... but if that's the goal, then using mozalloc to inject faults probably makes more sense. (And the JS code might already have some support?) Especially if the fuzzer can do something like unwind the stack and consult an external daemon to ensure that the fuzzer injects a fault for each allocation permutation based on the stack at most once, etc.
|
|
||
Comment 43•4 years ago
|
||
(I just received an autonag for this bug to prevent fuzzing from making progress, it seems the bot ignores the stalled keyword in this case)
|
|
Reporter | |
Comment 44•4 years ago
|
||
(In reply to Jens Stutte [:jstutte] from comment #43)
(I just received an autonag for this bug to prevent fuzzing from making progress, it seems the bot ignores the stalled keyword in this case)
suhaib: Is that expected?
|
||
Comment 45•4 years ago
|
||
(In reply to Tyson Smith [:tsmith] from comment #44)
(In reply to Jens Stutte [:jstutte] from comment #43)
(I just received an autonag for this bug to prevent fuzzing from making progress, it seems the bot ignores the stalled keyword in this case)
suhaib: Is that expected?
The bot does not exclude bugs with the stalled keyword. Based on comment 41, the bug is still a fuzzblocker, so it seems that removing the fuzzblocker tag is not an option. I filed an issue to exclude stalled bugs: https://github.com/mozilla/relman-auto-nag/issues/1743
|
|
||
Updated•4 years ago
|
|
|
||
Updated•3 years ago
|
|
|
Reporter | |
Comment 46•3 years ago
|
||
I am removing [fuzzblocker] because we no longer fuzz 32-bit android builds.
|
|
Reporter | |
Comment 47•3 years ago
|
||
Another thing worth noting that there was substantial drop in reports of this issue between 2022-10-31 and 2022-11-01 (we were fuzzing 32-bit builds at the time). The last build that was frequently reporting issues was 20221031-2db9822e6dd3. We have seen only 10 or so reports in the 6 weeks since.
|
|
||
Comment 48•3 years ago
|
||
(In reply to Tyson Smith [:tsmith] from comment #47)
Another thing worth noting that there was substantial drop in reports of this issue between 2022-10-31 and 2022-11-01 (we were fuzzing 32-bit builds at the time). The last build that was frequently reporting issues was 20221031-2db9822e6dd3. We have seen only 10 or so reports in the 6 weeks since.
I see bug 1797750 landing in that range. It seems to address various smaller JS issues which might have had some positive impact here?
|
||
Comment 49•3 years ago
|
||
Unlikely, as those fixes almost exclusively would impact test cases using ShadowRealms.
|
|
||
Updated•3 years ago
|
|
|
Reporter | |
Comment 50•3 years ago
•
|
||
I am unable to reproduce the issue with the attached test case.
The fuzzers are still reporting following assertion but infrequently (35 reports since May 1).
Assertion failure: false (OOM while building cycle collector graph), at /builds/worker/checkouts/gecko/xpcom/base/nsCycleCollector.cpp:2046
|
||
Comment 51•2 years ago
|
||
We are trying to close unactionable stalled bugs; given the lower frequency of the occurrance (and no longer on x86 android I imagine) we will close this out, but if it winds up where we can reproduce this in an environment that we can debug we could re-open it.
|
||
Comment 52•2 years ago
|
||
Since the bug is closed, the stalled keyword is now meaningless.
For more information, please visit BugBot documentation.
|
||
Updated•2 years ago
|
Description
•