Closed
Bug 1742381
Opened 3 years ago
Closed 3 years ago
Intermittent Assertion failure: refcount == 0 (There are `SharedImmutable[TwoByte]String`s outliving their associated cache! This always leads to use-after-free in the `~SharedImmutableString` destructor!), at SharedImmutableStringsCache.h:269
Categories
(Core :: JavaScript Engine, defect, P5)
Core
JavaScript Engine
Tracking
()
RESOLVED
FIXED
96 Branch
| Tracking | Status | |
|---|---|---|
| thunderbird_esr91 | --- | unaffected |
| firefox-esr91 | --- | unaffected |
| firefox94 | --- | unaffected |
| firefox95 | --- | unaffected |
| firefox96 | --- | fixed |
People
(Reporter: intermittent-bug-filer, Assigned: arai)
References
(Regression)
Details
(5 keywords)
Attachments
(1 file, 1 obsolete file)
Filed by: ncsoregi [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=358826918&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Shgha8cyRRSfTX4sDLSIWQ/runs/0/artifacts/public/logs/live_backing.log
[task 2021-11-22T10:30:43.872Z] 10:30:43 INFO - PID 6126 | Assertion failure: refcount == 0 (There are `SharedImmutable[TwoByte]String`s outliving their associated cache! This always leads to use-after-free in the `~SharedImmutableString` destructor!), at /builds/worker/checkouts/gecko/js/src/vm/SharedImmutableStringsCache.h:269
[task 2021-11-22T10:30:43.874Z] 10:30:43 INFO - PID 6126 | #01: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0xf7bd1c1]
[task 2021-11-22T10:30:43.876Z] 10:30:43 INFO - PID 6126 | #02: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0xf84d8dd]
[task 2021-11-22T10:30:43.877Z] 10:30:43 INFO - PID 6126 | #03: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0xf84d72c]
[task 2021-11-22T10:30:43.879Z] 10:30:43 INFO - PID 6126 | #04: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0xf84d578]
[task 2021-11-22T10:30:43.880Z] 10:30:43 INFO - PID 6126 | #05: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0xf8446e6]
[task 2021-11-22T10:30:43.881Z] 10:30:43 INFO - PID 6126 | #06: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0xf67a4c8]
[task 2021-11-22T10:30:43.882Z] 10:30:43 INFO - PID 6126 | #07: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x3c4a2be]
[task 2021-11-22T10:30:43.888Z] 10:30:43 INFO - PID 6126 | #08: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x59df0a0]
[task 2021-11-22T10:30:43.889Z] 10:30:43 INFO - PID 6126 | #09: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x5a7a54b]
[task 2021-11-22T10:30:43.890Z] 10:30:43 INFO - PID 6126 | #10: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x5a7a322]
[task 2021-11-22T10:30:43.892Z] 10:30:43 INFO - PID 6126 | #11: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x5a7ab85]
[task 2021-11-22T10:30:43.895Z] 10:30:43 INFO - PID 6126 | #12: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x3e1d955]
[task 2021-11-22T10:30:43.896Z] 10:30:43 INFO - PID 6126 | #13: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x3ee1337]
[task 2021-11-22T10:30:43.897Z] 10:30:43 INFO - PID 6126 | #14: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0xf10975d]
[task 2021-11-22T10:30:43.899Z] 10:30:43 INFO - PID 6126 | #15: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0x50360af]
[task 2021-11-22T10:30:43.900Z] 10:30:43 INFO - PID 6126 | #16: ???[/builds/worker/workspace/build/application/firefox/libxul.so +0xf10a007]
[task 2021-11-22T10:30:43.901Z] 10:30:43 INFO - PID 6126 | #17: ???[/builds/worker/workspace/build/application/firefox/firefox +0x1086ae]
[task 2021-11-22T10:30:43.906Z] 10:30:43 INFO - PID 6126 | #18: ???[/builds/worker/workspace/build/application/firefox/firefox +0x108ace]
[task 2021-11-22T10:30:43.912Z] 10:30:43 INFO - PID 6126 | #19: __libc_start_main[/lib/x86_64-linux-gnu/libc.so.6 +0x21b97]
[task 2021-11-22T10:30:43.913Z] 10:30:43 INFO - PID 6126 | #20: ???[/builds/worker/workspace/build/application/firefox/firefox +0x5779d]
[task 2021-11-22T10:30:43.917Z] 10:30:43 INFO - PID 6126 | #21: ??? (???:???)
[task 2021-11-22T10:30:43.918Z] 10:30:43 INFO - PID 6126 | AddressSanitizer:DEADLYSIGNAL
[task 2021-11-22T10:30:43.919Z] 10:30:43 INFO - PID 6126 | =================================================================
[task 2021-11-22T10:30:43.919Z] 10:30:43 ERROR - PID 6126 | ==6699==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f6b3944f1ed bp 0x7ffd3331e270 sp 0x7ffd3331e260 T0)
[task 2021-11-22T10:30:43.925Z] 10:30:43 INFO - PID 6126 | ==6699==The signal is caused by a WRITE memory access.
[task 2021-11-22T10:30:43.926Z] 10:30:43 INFO - PID 6126 | ==6699==Hint: address points to the zero page.
[task 2021-11-22T10:30:44.143Z] 10:30:44 INFO - PID 6126 | -----------------------------------------------------
[task 2021-11-22T10:30:45.541Z] 10:30:45 INFO - PID 6126 | -----------------------------------------------------
[task 2021-11-22T10:30:45.681Z] 10:30:45 INFO - PID 6126 | #0 0x7f6b3944f1ed in ~StringBox /builds/worker/checkouts/gecko/js/src/vm/SharedImmutableStringsCache.h:265:7
[task 2021-11-22T10:30:45.682Z] 10:30:45 INFO - PID 6126 | #1 0x7f6b3944f1ed in js_delete<js::SharedImmutableStringsCache::StringBox> /builds/worker/workspace/obj-build/dist/include/js/Utility.h:552:9
[task 2021-11-22T10:30:45.683Z] 10:30:45 INFO - PID 6126 | #2 0x7f6b3944f1ed in JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox>::operator()(js::SharedImmutableStringsCache::StringBox const*) /builds/worker/workspace/obj-build/dist/include/js/Utility.h:625:35
[task 2021-11-22T10:30:45.684Z] 10:30:45 INFO - PID 6126 | #3 0x7f6b394df8dc in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:305:7
[task 2021-11-22T10:30:45.685Z] 10:30:45 INFO - PID 6126 | #4 0x7f6b394df8dc in ~UniquePtr /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:253:18
[task 2021-11-22T10:30:45.686Z] 10:30:45 INFO - PID 6126 | #5 0x7f6b394df8dc in destroyStoredT /builds/worker/workspace/obj-build/dist/include/mozilla/HashTable.h:1059:11
[task 2021-11-22T10:30:45.688Z] 10:30:45 INFO - PID 6126 | #6 0x7f6b394df8dc in mozilla::detail::HashTable<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> > const, mozilla::HashSet<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> >, js::SharedImmutableStringsCache::Hasher, js::SystemAllocPolicy>::SetHashPolicy, js::SystemAllocPolicy>::destroyTable(js::SystemAllocPolicy&, char*, unsigned int)::'lambda'(mozilla::detail::EntrySlot<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> > const> const&)::operator()(mozilla::detail::EntrySlot<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> > const> const&) const /builds/worker/workspace/obj-build/dist/include/mozilla/HashTable.h:1670:25
[task 2021-11-22T10:30:45.690Z] 10:30:45 INFO - PID 6126 | #7 0x7f6b394df72b in void mozilla::detail::HashTable<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> > const, mozilla::HashSet<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> >, js::SharedImmutableStringsCache::Hasher, js::SystemAllocPolicy>::SetHashPolicy, js::SystemAllocPolicy>::forEachSlot<mozilla::detail::HashTable<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> > const, mozilla::HashSet<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> >, js::SharedImmutableStringsCache::Hasher, js::SystemAllocPolicy>::SetHashPolicy, js::SystemAllocPolicy>::destroyTable(js::SystemAllocPolicy&, char*, unsigned int)::'lambda'(mozilla::detail::EntrySlot<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> > const> const&)>(char*, unsigned int, mozilla::detail::HashTable<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> > const, mozilla::HashSet<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> >, js::SharedImmutableStringsCache::Hasher, js::SystemAllocPolicy>::SetHashPolicy, js::SystemAllocPolicy>::destroyTable(js::SystemAllocPolicy&, char*, unsigned int)::'lambda'(mozilla::detail::EntrySlot<mozilla::UniquePtr<js::SharedImmutableStringsCache::StringBox, JS::DeletePolicy<js::SharedImmutableStringsCache::StringBox> > const> const&)&&) /builds/worker/workspace/obj-build/dist/include/mozilla/HashTable.h:1204:7
[task 2021-11-22T10:30:45.691Z] 10:30:45 INFO - PID 6126 | #8 0x7f6b394df577 in destroyTable /builds/worker/workspace/obj-build/dist/include/mozilla/HashTable.h:1668:5
[task 2021-11-22T10:30:45.692Z] 10:30:45 INFO - PID 6126 | #9 0x7f6b394df577 in ~HashTable /builds/worker/workspace/obj-build/dist/include/mozilla/HashTable.h:1703:7
[task 2021-11-22T10:30:45.693Z] 10:30:45 INFO - PID 6126 | #10 0x7f6b394df577 in ~HashSet /builds/worker/workspace/obj-build/dist/include/mozilla/HashTable.h:435:7
[task 2021-11-22T10:30:45.693Z] 10:30:45 INFO - PID 6126 | #11 0x7f6b394df577 in ~Inner /builds/worker/checkouts/gecko/js/src/vm/SharedImmutableStringsCache.h:348:43
[task 2021-11-22T10:30:45.694Z] 10:30:45 INFO - PID 6126 | #12 0x7f6b394df577 in ~ExclusiveData /builds/worker/checkouts/gecko/js/src/threading/ExclusiveData.h:85:7
[task 2021-11-22T10:30:45.695Z] 10:30:45 INFO - PID 6126 | #13 0x7f6b394df577 in js_delete<js::ExclusiveData<js::SharedImmutableStringsCache::Inner> > /builds/worker/workspace/obj-build/dist/include/js/Utility.h:552:9
[task 2021-11-22T10:30:45.696Z] 10:30:45 INFO - PID 6126 | #14 0x7f6b394df577 in js::SharedImmutableStringsCache::~SharedImmutableStringsCache() /builds/worker/checkouts/gecko/js/src/vm/SharedImmutableStringsCache.h:210:7
[task 2021-11-22T10:30:45.697Z] 10:30:45 INFO - PID 6126 | #15 0x7f6b394d66e5 in ~MaybeStorage /builds/worker/workspace/obj-build/dist/include/mozilla/Maybe.h:268:25
[task 2021-11-22T10:30:45.697Z] 10:30:45 INFO - PID 6126 | #16 0x7f6b394d66e5 in JSRuntime::~JSRuntime() /builds/worker/checkouts/gecko/js/src/vm/Runtime.cpp:190:1
[task 2021-11-22T10:30:45.698Z] 10:30:45 INFO - PID 6126 | #17 0x7f6b3930c4c7 in js_delete_poison<JSRuntime> /builds/worker/workspace/obj-build/dist/include/js/Utility.h:560:9
[task 2021-11-22T10:30:45.699Z] 10:30:45 INFO - PID 6126 | #18 0x7f6b3930c4c7 in js::DestroyContext(JSContext*) /builds/worker/checkouts/gecko/js/src/vm/JSContext.cpp:246:3
[task 2021-11-22T10:30:45.700Z] 10:30:45 INFO - PID 6126 | #19 0x7f6b2d8dc2bd in mozilla::CycleCollectedJSContext::~CycleCollectedJSContext() /builds/worker/checkouts/gecko/xpcom/base/CycleCollectedJSContext.cpp:110:3
[task 2021-11-22T10:30:45.701Z] 10:30:45 INFO - PID 6126 | #20 0x7f6b2f67109f in XPCJSContext::~XPCJSContext() /builds/worker/checkouts/gecko/js/xpconnect/src/XPCJSContext.cpp:1106:1
[task 2021-11-22T10:30:45.701Z] 10:30:45 INFO - PID 6126 | #21 0x7f6b2f70c54a in nsXPConnect::~nsXPConnect() /builds/worker/checkouts/gecko/js/xpconnect/src/nsXPConnect.cpp:129:3
[task 2021-11-22T10:30:45.702Z] 10:30:45 INFO - PID 6126 | #22 0x7f6b2f70c321 in nsXPConnect::Release() /builds/worker/checkouts/gecko/js/xpconnect/src/nsXPConnect.cpp:57:1
[task 2021-11-22T10:30:45.703Z] 10:30:45 INFO - PID 6126 | #23 0x7f6b2f70cb84 in nsXPConnect::ReleaseXPConnectSingleton() /builds/worker/checkouts/gecko/js/xpconnect/src/nsXPConnect.cpp:164:5
[task 2021-11-22T10:30:45.704Z] 10:30:45 INFO - PID 6126 | #24 0x7f6b2daaf954 in nsComponentManagerImpl::Shutdown() /builds/worker/checkouts/gecko/xpcom/components/nsComponentManager.cpp:852:3
[task 2021-11-22T10:30:45.705Z] 10:30:45 INFO - PID 6126 | #25 0x7f6b2db73336 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:725:55
[task 2021-11-22T10:30:45.705Z] 10:30:45 INFO - PID 6126 | #26 0x7f6b38d9b75c in XRE_TermEmbedding() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:218:3
[task 2021-11-22T10:30:45.706Z] 10:30:45 INFO - PID 6126 | #27 0x7f6b2ecc80ae in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/checkouts/gecko/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2021-11-22T10:30:45.707Z] 10:30:45 INFO - PID 6126 | #28 0x7f6b38d9c006 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:754:16
[task 2021-11-22T10:30:45.708Z] 10:30:45 INFO - PID 6126 | #29 0x563e2b3ac6ad in content_process_main(mozilla::Bootstrap*, int, char**) /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:57:28
[task 2021-11-22T10:30:45.709Z] 10:30:45 INFO - PID 6126 | #30 0x563e2b3acacd in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:327:18
[task 2021-11-22T10:30:45.709Z] 10:30:45 INFO - PID 6126 | #31 0x7f6b4fe22b96 in __libc_start_main /tmp/glibc/csu/../csu/libc-start.c:310
[task 2021-11-22T10:30:45.710Z] 10:30:45 INFO - PID 6126 | #32 0x563e2b2fb79c in _start (/builds/worker/workspace/build/application/firefox/firefox+0x5779c)
[task 2021-11-22T10:30:45.711Z] 10:30:45 INFO - PID 6126 | AddressSanitizer can not provide additional info.
[task 2021-11-22T10:30:45.712Z] 10:30:45 INFO - PID 6126 | SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/js/src/vm/SharedImmutableStringsCache.h:265:7 in ~StringBox
[task 2021-11-22T10:30:45.713Z] 10:30:45 INFO - PID 6126 | ==6699==ABORTING
|
||
Comment 1•3 years ago
|
||
This is deep in main thread shutdown, so I assume it isn't really exploitable.
Keywords: csectype-uaf,
sec-moderate
|
||
Updated•3 years ago
|
Group: core-security → javascript-core-security
|
|
||
Comment 2•3 years ago
|
||
Arai, could this be related to the stencil shutdown changes you made recently? I have no idea if these caches are related, but they are in the same ballpark of teardown of JS data caches.
Flags: needinfo?(arai.unmht)
|
Assignee | |
Comment 3•3 years ago
|
||
Yes, given that this code is after fast shutdown with default configuration, this crash or UAF doesn't happen in release binary,
Then, so far I cannot reproduce the error locally.
There's indeed StringBox reference used in Stencil, that's held by ScriptPreloader singleton, but I don't see lifetime change in the patch.
class ScriptPreloader : ... {
...
class CachedStencil : public LinkedListElement<CachedStencil> {
...
RefPtr<JS::Stencil> mStencil;
...
};
...
using ScriptHash = nsClassHashtable<nsCStringHashKey, CachedStencil>;
...
ScriptHash mScripts;
...
};
namespace JS {
...
using Stencil = js::frontend::CompilationStencil;
...
}
struct CompilationStencil {
...
RefPtr<ScriptSource> source;
...
};
class ScriptSource {
...
template <typename Unit>
class UncompressedData {
typename SourceTypeTraits<Unit>::SharedImmutableString string_;
...
};
...
template <typename Unit>
struct CompressedData {
...
SharedImmutableString raw;
};
...
SharedImmutableString filename_;
...
SharedImmutableString introducerFilename_;
...
SharedImmutableTwoByteString displayURL_;
SharedImmutableTwoByteString sourceMapURL_;
...
};
class SharedImmutableString {
...
mutable SharedImmutableStringsCache::StringBox* box_;
...
};
SharedImmutableString::~SharedImmutableString() {
if (!box_) {
return;
}
auto locked = box_->cache_->lock();
MOZ_ASSERT(box_->refcount > 0);
box_->refcount--;
...
}
The crash happens at line 725 (nsComponentManagerImpl::Shutdown call) in ShutdownXPCOM function.
The bug 1738282 patch moved the ScriptPreloader singleton destructor call from line 650 (XPCOMShutdownFinal phase) to line 718 (ScriptPreloader::DeleteSingleton call).
Both are before line 725, so the StringBox refcount shouldn't be affected by that.
nsresult ShutdownXPCOM(nsIServiceManager* aServMgr) {
...
mozilla::KillClearOnShutdown(ShutdownPhase::XPCOMShutdownFinal); // line 650
...
mozilla::ScriptPreloader::DeleteSingleton(); // line 718
...
if (nsComponentManagerImpl::gComponentManager) {
rv = (nsComponentManagerImpl::gComponentManager)->Shutdown(); // line 725
We could add some diagnostic print to StringBox destructor, to see what the content is.
If this crash happens again in automation, the print will tell us where the string comes from.
|
|
Assignee | |
Comment 4•3 years ago
|
||
this doesn't happen locally after running wpt wdspec for 5 hours (50 cycles)
I'll try adding debug print and see the result on automation.
|
|
Assignee | |
Updated•3 years ago
|
Flags: needinfo?(arai.unmht)
Keywords: leave-open
|
|
Assignee | |
Comment 5•3 years ago
|
||
|
||
Comment 6•3 years ago
•
|
||
The piece of info we are missing where the late reference is finally cleaned up. I've put together a simple experiment that leaks these strings instead of crashing, and then when they are finally freed will crash and give us that stack. Running the wdspec jobs on try now: https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&revision=e60360313cf75c127c5e7649f6b46a55590371c9
|
|
||
Comment 7•3 years ago
|
||
An interesting detail from my experiment is that I am now leaking NewRunnableMethodfrommozilla::ScriptPreloader::OffThreadDecodeCallback`.
One change from moving the ScriptPreloader::Cleanup call from xpcom-shutdown to later is that we mess with the off-thread cleanup. I'm not sure what best answer is, but this seems directly due to Bug 1738282.
One option might be to put back the xpcom-shutdown but only do the thread join (which after early startup has no threads left) and then only defer the mScripts stuff (which can be just the implicit destructor).
|
||
Updated•3 years ago
|
Attachment #9252291 -
Attachment is obsolete: true
|
|
Assignee | |
Comment 8•3 years ago
|
||
|
|
Assignee | |
Updated•3 years ago
|
Keywords: leave-open
|
|
Assignee | |
Updated•3 years ago
|
Assignee: nobody → arai.unmht
Status: NEW → ASSIGNED
|
|
Assignee | |
Updated•3 years ago
|
status-firefox94:
--- → unaffected
status-firefox95:
--- → unaffected
status-firefox96:
--- → affected
status-firefox-esr91:
--- → unaffected
Regressed by: 1738282
|
||
Updated•3 years ago
|
Has Regression Range: --- → yes
Keywords: regression
|
|
Assignee | |
Comment 9•3 years ago
|
||
|
||
Comment 10•3 years ago
|
||
Group: javascript-core-security → core-security-release
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 96 Branch
|
|
||
Comment 11•3 years ago
|
||
Looks like the crashes have dropped off so I think we fixed the right thing.
|
||
Updated•3 years ago
|
Flags: qe-verify-
|
|
||
Updated•3 years ago
|
Group: core-security-release
You need to log in
before you can comment on or make changes to this bug.
Description
•