ThreadSanitizer: heap-use-after-free [@ Equals] in FrameProperties.h, due to our lack-of-a-null-check causing compiler to optimize away an earlier array-bounds-check
Categories
(Core :: Layout, defect)
Tracking
()
People
(Reporter: tsmith, Assigned: dholbert)
References
(Blocks 2 open bugs, Regression)
Details
(6 keywords, Whiteboard: [bugmon:bisected,confirmed][pp3][adv-esr140.10.1+r][adv-main150.0.1+r])
Crash Data
Attachments
(7 files)
|
363 bytes,
text/html
|
Details | |
|
38.68 KB,
text/plain
|
Details | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
tjr
:
sec-approval+
|
Details | Review |
|
48 bytes,
text/x-phabricator-request
|
phab-bot
:
approval-mozilla-beta+
|
Details | Review |
|
48 bytes,
text/x-phabricator-request
|
phab-bot
:
approval-mozilla-esr140+
|
Details | Review |
|
48 bytes,
text/x-phabricator-request
|
phab-bot
:
approval-mozilla-release+
|
Details | Review |
Found while fuzzing m-c 20260330-0b76dd137a61 (--enable-thread-sanitizer --enable-fuzzing)
To reproduce via Grizzly Replay:
$ pip install fuzzfetch grizzly-framework --upgrade
$ python -m fuzzfetch -t --fuzzing -n firefox
$ python -m grizzly.replay.bugzilla ./firefox/firefox <bugid>
NOTE: I have only been able to reproduce this with TSan not ASan
WARNING: ThreadSanitizer: heap-use-after-free (pid=141176)
Read of size 8 at 0x722000293b88 by main thread:
#0 Equals /builds/worker/checkouts/gecko/layout/base/FrameProperties.h:345:16 (libxul.so+0xa18ec87) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#1 Equals<const mozilla::FrameProperties::PropertyValue, const mozilla::FramePropertyDescriptorUntyped *const> /builds/worker/workspace/obj-build/dist/include/nsTArray.h:832:24 (libxul.so+0xa18ec87)
#2 ApplyIf<const mozilla::FramePropertyDescriptorUntyped *, mozilla::FrameProperties::PropertyComparator, (lambda at /builds/worker/checkouts/gecko/layout/base/FrameProperties.h:358:7), (lambda at /builds/worker/checkouts/gecko/layout/base/FrameProperties.h:364:7)> /builds/worker/workspace/obj-build/dist/include/nsTArray.h:2027:16 (libxul.so+0xa18ec87)
#3 GetInternal /builds/worker/checkouts/gecko/layout/base/FrameProperties.h:356:22 (libxul.so+0xa18ec87)
#4 Get<nsContainerFrame> /builds/worker/checkouts/gecko/layout/base/FrameProperties.h:197:18 (libxul.so+0xa18ec87)
#5 GetProperty<nsContainerFrame> /builds/worker/checkouts/gecko/layout/generic/nsIFrame.h:4445:24 (libxul.so+0xa18ec87)
#6 nsInlineFrame::UpdateStyleOfOwnedAnonBoxesForIBSplit(mozilla::ServoRestyleState&) /builds/worker/checkouts/gecko/layout/generic/nsInlineFrame.cpp:927:40 (libxul.so+0xa18ec87)
#7 nsIFrame::DoUpdateStyleOfOwnedAnonBoxes(mozilla::ServoRestyleState&) /builds/worker/checkouts/gecko/layout/generic/nsIFrame.cpp:12073:42 (libxul.so+0xa18e9a3) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#8 UpdateStyleOfOwnedAnonBoxes /builds/worker/checkouts/gecko/layout/generic/nsIFrame.h:4268:7 (libxul.so+0x9ea48ed) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#9 mozilla::RestyleManager::ProcessPostTraversal(mozilla::dom::Element*, mozilla::ServoRestyleState&, mozilla::ServoPostTraversalFlags) /builds/worker/checkouts/gecko/layout/style/RestyleManager.cpp:3012:19 (libxul.so+0x9ea48ed)
#10 mozilla::RestyleManager::ProcessPostTraversal(mozilla::dom::Element*, mozilla::ServoRestyleState&, mozilla::ServoPostTraversalFlags) /builds/worker/checkouts/gecko/layout/style/RestyleManager.cpp:2994:32 (libxul.so+0x9ea4776) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#11 mozilla::RestyleManager::ProcessPostTraversal(mozilla::dom::Element*, mozilla::ServoRestyleState&, mozilla::ServoPostTraversalFlags) /builds/worker/checkouts/gecko/layout/style/RestyleManager.cpp:2994:32 (libxul.so+0x9ea4776) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#12 mozilla::RestyleManager::ProcessPostTraversal(mozilla::dom::Element*, mozilla::ServoRestyleState&, mozilla::ServoPostTraversalFlags) /builds/worker/checkouts/gecko/layout/style/RestyleManager.cpp:2994:32 (libxul.so+0x9ea4776) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#13 mozilla::RestyleManager::DoProcessPendingRestyles(mozilla::ServoTraversalFlags) /builds/worker/checkouts/gecko/layout/style/RestyleManager.cpp:3210:28 (libxul.so+0x9ea59a9) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#14 mozilla::RestyleManager::ProcessPendingRestyles() /builds/worker/checkouts/gecko/layout/style/RestyleManager.cpp:3328:3 (libxul.so+0x9ea66d2) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#15 mozilla::PresShell::DoFlushPendingNotifications(mozilla::ChangesToFlush) /builds/worker/checkouts/gecko/layout/base/PresShell.cpp:4489:37 (libxul.so+0x9f780fa) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#16 FlushPendingNotifications /builds/worker/workspace/obj-build/dist/include/mozilla/PresShell.h:1528:5 (libxul.so+0x60c7d91) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
...
Previous write of size 8 at 0x722000293b88 by thread T29:
#0 free /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:741:3 (firefox-bin+0xd654a) (BuildId: b2638e10e9e12dac22015efee0a348aa876dc757)
#1 operator delete /builds/worker/workspace/obj-build/dist/include/mozilla/cxxalloc.h:64:10 (libxul.so+0x49c0e72) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#2 mojo::core::ports::UserMessageEvent::~UserMessageEvent() /builds/worker/checkouts/gecko/ipc/chromium/src/mojo/core/ports/event.cc:179:37 (libxul.so+0x49c0e72)
#3 operator() /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/unique_ptr.h:85:2 (libxul.so+0x4a38b75) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#4 ~unique_ptr /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/10/../../../../include/c++/10/bits/unique_ptr.h:361:4 (libxul.so+0x4a38b75)
#5 mozilla::ipc::NodeController::ContactRemotePeer(mojo::core::ports::NodeName const&, std::unique_ptr<mojo::core::ports::Event, std::default_delete<mojo::core::ports::Event>>) /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:333:5 (libxul.so+0x4a38b75)
#6 mozilla::ipc::NodeController::ForwardEvent(mojo::core::ports::NodeName const&, std::unique_ptr<mojo::core::ports::Event, std::default_delete<mojo::core::ports::Event>>) /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:408:5 (libxul.so+0x4a39413) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#7 mojo::core::ports::Node::SendUserMessageInternal(mojo::core::ports::PortRef const&, std::unique_ptr<mojo::core::ports::UserMessageEvent, std::default_delete<mojo::core::ports::UserMessageEvent>>*) /builds/worker/checkouts/gecko/ipc/chromium/src/mojo/core/ports/node.cc:1271:16 (libxul.so+0x49cb7ff) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#8 mojo::core::ports::Node::SendUserMessage(mojo::core::ports::PortRef const&, std::unique_ptr<mojo::core::ports::UserMessageEvent, std::default_delete<mojo::core::ports::UserMessageEvent>>) /builds/worker/checkouts/gecko/ipc/chromium/src/mojo/core/ports/node.cc:379:12 (libxul.so+0x49cb45e) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#9 mozilla::ipc::NodeController::SendUserMessage(mojo::core::ports::PortRef const&, std::unique_ptr<IPC::Message, std::default_delete<IPC::Message>>) /builds/worker/checkouts/gecko/ipc/glue/NodeController.cpp:151:19 (libxul.so+0x4a333da) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#10 mozilla::ipc::PortLink::SendMessage(std::unique_ptr<IPC::Message, std::default_delete<IPC::Message>>) /builds/worker/checkouts/gecko/ipc/glue/MessageLink.cpp:128:16 (libxul.so+0x4a330c1) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#11 mozilla::ipc::MessageChannel::SendMessageToLink(std::unique_ptr<IPC::Message, std::default_delete<IPC::Message>>) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:761:10 (libxul.so+0x4a28bb6) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#12 mozilla::ipc::MessageChannel::Send(std::unique_ptr<IPC::Message, std::default_delete<IPC::Message>>, long*) /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:730:3 (libxul.so+0x4a27f37) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#13 mozilla::ipc::IProtocol::ChannelSend(std::unique_ptr<IPC::Message, std::default_delete<IPC::Message>>, long*) /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:487:22 (libxul.so+0x4a417e7) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#14 mozilla::PRemoteLazyInputStreamChild::SendStreamNeeded(unsigned long const&, unsigned long const&, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_fixed<16ul, 8ul>>, fu2::abi_400::detail::property<false, false, void (mozilla::Maybe<mozilla::ipc::IPCStream>&&)>>&&, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_fixed<16ul, 8ul>>, fu2::abi_400::detail::property<false, false, void (mozilla::ipc::ResponseRejectReason)>>&&) /builds/worker/workspace/obj-build/ipc/ipdl/PRemoteLazyInputStreamChild.cpp:130:9 (libxul.so+0x7be5a35) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
...
Thread T29 'RemoteLzyStream' (tid=141227, running) created by main thread at:
#0 pthread_create /builds/worker/fetches/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1041:3 (firefox-bin+0xd798a) (BuildId: b2638e10e9e12dac22015efee0a348aa876dc757)
#1 _PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:429:10 (libnspr4.so+0x425fe) (BuildId: 8adc702b321e97005c58eb14bcc147b1288ad047)
#2 PR_CreateThread /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:496:10 (libnspr4.so+0x37974) (BuildId: 8adc702b321e97005c58eb14bcc147b1288ad047)
#3 nsThread::Init(nsTSubstring<char> const&) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:623:20 (libxul.so+0x3eba8cb) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#4 nsThreadManager::NewNamedThread(nsTSubstring<char> const&, nsIThreadManager::ThreadCreationOptions, nsIThread**) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadManager.cpp:613:22 (libxul.so+0x3ec192a) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#5 NS_NewNamedThread(nsTSubstring<char> const&, nsIThread**, already_AddRefed<nsIRunnable>, nsIThreadManager::ThreadCreationOptions) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:179:57 (libxul.so+0x3ec9a25) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#6 NS_NewNamedThread<16UL> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:85:10 (libxul.so+0x7bdb38f) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#7 mozilla::RemoteLazyInputStreamThread::GetOrCreate() /builds/worker/checkouts/gecko/dom/file/ipc/RemoteLazyInputStreamThread.cpp:50:19 (libxul.so+0x7bdb38f)
#8 mozilla::BindChildActor(nsID, mozilla::ipc::Endpoint<mozilla::PRemoteLazyInputStreamChild>) /builds/worker/checkouts/gecko/dom/file/ipc/RemoteLazyInputStream.cpp:160:7 (libxul.so+0x7bd7a76) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#9 mozilla::RemoteLazyInputStream::IPCRead(IPC::MessageReader*) /builds/worker/checkouts/gecko/dom/file/ipc/RemoteLazyInputStream.cpp:1426:7 (libxul.so+0x7bde672) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
#10 Read /builds/worker/checkouts/gecko/dom/file/ipc/RemoteLazyInputStream.cpp:1459:14 (libxul.so+0x7be2d1d) (BuildId: b86fd9fd6ac6d08074d6388b848ffcfc00c2338c)
| Reporter | ||
Comment 1•5 months ago
|
||
Comment 2•5 months ago
|
||
Verified bug as reproducible on mozilla-central 20260401160113-a5514d76e0e7.
Unable to bisect testcase (Testcase reproduces on start build!):
Start: 526b3d0ba7ebe24e98aad3e1535e586927651821 (20250402162225)
End: 0b76dd137a61420640d9276c563698c3b3eff72a (20260330213624)
BuildFlags: BuildFlags(asan=False, tsan=True, debug=False, fuzzing=True, coverage=False, valgrind=False, no_opt=False, fuzzilli=False, nyx=False, searchfox=False, afl=False)
| Assignee | ||
Updated•5 months ago
|
| Assignee | ||
Comment 3•5 months ago
|
||
Initially assuming this is S2 given this is a heap-use-after-free. A pernosco trace will hopefully make this easier to reason about.
| Assignee | ||
Comment 4•5 months ago
|
||
(ni=me to remind myself to investigate after a pernosco trace appears)
Comment 5•5 months ago
|
||
:dholbert, unfortunately we don't have TSan no-op builds so bugmon won't be able to gather a trace. The best we could do is a TSan opt build where most of the variables will be optimized away.
Updated•5 months ago
|
Comment 6•5 months ago
|
||
(Drive-by investigation)
Intesrestingly, the testcase crashes reliably if in debug build, I break on nsInlineFrame::UpdateStyleOfOwnedAnonBoxesForIBSplit, then continue.
In rr, refreshing enough times will result in a crash as well.
The crash is here. Seems that we assume that there's a trailing inline frame in an IB-split set of frames, but we don't. This makes sense because looking at the frame tree, blockFrame is:
Block(data)(1)@7fd42c3cf738 parent=7fd42c3cf010 next-in-flow=7fd42c18e030 IBSplitPrevSibling=7fd42c3cf2a8 (x=0, y=1286, w=60480, h=564) [content=7fd426a105e0][cs=7fd42c3cc608][MozBlockInsideInlineWrapper] <
Note the lack of inline IBSplit sibling. next-in-flow is in the overflow container:
Block(html)(-1)@7fd42c18e338 parent=7fd42c3ceb50 prev-in-flow=7fd42c3ce9c0 (x=62400, y=0, w=61440, h=0) ink-overflow=(x=0, y=0, w=491040, h=2034) scr-overflow=(x=0, y=0, w=491040, h=2034) [content=7fd426a04c10][cs=7fd42c3cd008][MozColumnContent] <
OverflowContainersList@7fd42c18f6a8 <
ColumnSetWrapper(body)(1)@7fd42c18e108 parent=7fd42c18e338 prev-in-flow=7fd42c3ced20 (x=480, y=0, w=60480, h=0) ink-overflow=(x=0, y=0, w=490560, h=2034) scr-overflow=(x=0, y=0, w=490560, h=2034) [content=7fd426a10040][cs=7fd42c3cbc08] <
OverflowContainersList@7fd42c18f698 <
ColumnSet(body)(1)@7fd42c18f5d0 parent=7fd42c18e108 prev-in-flow=7fd42c3cede8 (x=0, y=0, w=61440, h=2034) ink-overflow=(x=0, y=0, w=490560, h=2034) scr-overflow=(x=0, y=0, w=490560, h=2034) [content=7fd426a10040][cs=7fd42c3cd508][MozColumnSet] <
Block(body)(1)@7fd42c3cfc78 parent=7fd42c18f5d0 prev-in-flow=7fd42c3cec58 (x=0, y=0, w=61440, h=0) ink-overflow=(x=0, y=0, w=490560, h=184) scr-overflow=(x=0, y=0, w=490560, h=184) [content=7fd426a10040][cs=7fd42c3ccb08][MozColumnContent] <
OverflowContainersList@7fd42c18e020 <
GridContainer(dl id=a)(0)@7fd42c3cff30 parent=7fd42c3cfc78 prev-in-flow=7fd42c3cef40 (x=0, y=0, w=60480, h=0) ink-overflow=(x=0, y=0, w=490560, h=184) scr-overflow=(x=0, y=0, w=490560, h=184) [content=7fd426a103a0][cs=7fd42c346a08] <
ColumnSetWrapper(dt id=b)(1)@7fd42c3cfa48 parent=7fd42c3cff30 prev-in-flow=7fd42c3cf0d8 (x=0, y=0, w=60480, h=184) ink-overflow=(x=0, y=0, w=490560, h=184) scr-overflow=(x=0, y=0, w=490560, h=184) [content=7fd426a104c0][cs=7fd42c348008] <
line@7fd42c3cfc28 count=1 state=block,clean,prevmarginclean,not-impacted,not-wrapped,no-break,clear-before:none,clear-after:none (x=0, y=0, w=490560, h=184) <
ColumnSet(dt id=b)(1)@7fd42c3cfd50 parent=7fd42c3cfa48 prev-in-flow=7fd42c3cf1a0 (x=0, y=0, w=490560, h=184) [content=7fd426a104c0][cs=7fd42c3cde08][MozColumnSet] <
Block(dt id=b)(1)@7fd42c3cfb60 parent=7fd42c3cfd50 next=7fd42c18e630 prev-in-flow=7fd42c3cf010 next-in-flow=7fd42c18e630 (x=0, y=0, w=60480, h=184) [content=7fd426a104c0][cs=7fd42c3cda08][MozColumnContent] <
line@7fd42c3cfb10 count=1 state=block,clean,prevmarginclean,not-impacted,not-wrapped,no-break,clear-before:none,clear-after:none (x=0, y=0, w=60480, h=184) <
Block(data)(1)@7fd42c18e030 parent=7fd42c3cfb60 prev-in-flow=7fd42c3cf738 next-in-flow=7fd42c18e220 (x=0, y=0, w=60480, h=184) [content=7fd426a105e0][cs=7fd42c3cc608][MozBlockInsideInlineWrapper] <
... Makes sense given:
* {
height: 5vh;
columns: 1 1px;
}
Should be noted that removing the assert temporarily quickly results in a null pointer deref anyway.
OTOH, the TSAN doesn't seem like it's pointing to the same thing. We could add guarded returns within nsInlineFrame::UpdateStyleOfOwnedAnonBoxesForIBSplit, as the assumption it makes doesn't seem to hold when things end up in overflow containers, but we may just mask this issue.
Here's a pernosco trace of the crash, (Not the TSAN behaviour!)
Comment 7•5 months ago
|
||
| Assignee | ||
Comment 8•5 months ago
|
||
Looking at the TSAN issue in comment 0 a bit closer, it's unlikely to be a UAF in the way that TSAN is reporting it (but really might be a wild-pointer-read, or a UAF somewhere slightly earlier that doesn't get detected).
Specifically: in the "Previous write" section of comment 0, some IPC code deallocates a UserMessageEvent object (which is a type that has zero relation to layout or reflow code). And then apparently at some point later on, our reflow code reads that same memory in a FrameProperties lookup. That doesn't make sense; that read must have been some sort of coincidence, with a wild-pointer-read. (still concerning, but the badness is a perhaps a bit earlier) A typical UAF would involve us dereferencing a dangling pointer to the deleted object -- but that's clearly not the case here -- FrameProperties has no connection to UserMessageEvent and wouldn't have any pointers to such an object. So how did this happen?
It's conceivable that our FrameProperties object here is filled with garbage data for some reason, and one of those garbage values just-so-happened to be the address of this deleted UserMessageEvent...
I suspect TSAN is indeed hitting the same crash that dshin observed in comment 6. Probably we end up having a frame (or a dynamically allocated frame-properties object) that's been deallocated, such that our nsIFrame* nextInline = blockFrame->GetProperty(nsIFrame::IBSplitSibling()); invocation is working with garbage data to do the property lookup.
(In reply to David Shin[:dshin] from comment #6)
The crash is here. Seems that we assume that there's a trailing inline frame in an IB-split set of frames, but we don't.
[...]
Should be noted that removing the assert temporarily quickly results in a null pointer deref anyway.
Maybe getting a nullptr here (causing us to fail the assertion or null-pointer-deref) is the best-case scenario, and weirder stuff can happen depending on the nature of the garbage that we find...
| Assignee | ||
Comment 9•5 months ago
|
||
(In reply to Daniel Holbert [:dholbert] from comment #8)
I suspect TSAN is indeed hitting the same crash that dshin observed in comment 6. Probably we end up having a frame (or a dynamically allocated frame-properties object) that's been deallocated, such that our
nsIFrame* nextInline = blockFrame->GetProperty(nsIFrame::IBSplitSibling());invocation is working with garbage data to do the property lookup.
[...]Maybe getting a nullptr here (causing us to fail the assertion or null-pointer-deref) is the best-case scenario, and weirder stuff can happen depending on the nature of the garbage that we find...
...though, if this^ were the explanation, then ASAN should surely be able to hit UAFs with this testcase too (but per comment 0, it does not).
And poking around in the pernoso trace, it looks like we're operating with a perfectly-fine frame and a perfectly-fine property-table, which just lacks a value for IBSplitSibling -- i.e. the property table legitimately returns null, rather than garbage.
So I'm back to not really undersatnding the TSAN issue...
Tyson: if you're able to still reproduce this, would you mind sharing a few different logs (a few different copies of comment 0's data, from different instances of the crash)? I'm curious if there are any patterns to the addresses and to the "Previous write" struct (which seems superficially unrelated right now).
(I tried to answer my own question using the fuzzfetch/grizzly commands in comment 0, but for some reason the build insta-crashes on my machine before even loading a testcase), with WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) and Cycle in lock order graph:)
| Assignee | ||
Comment 10•5 months ago
|
||
Side note, emilio has some work-in-progress patches to greatly simplify how we handle block-inside-inline splits (IB Splits). There's a decent chance that that patch will avoid the assertion/crash that dshin mentioned in comment 6.
That might help here, but the timeline for that is a bit hazy; and it's also not clear if the issues we're hitting with TSAN (the most concerning but also least-well-understood part of this bug) would be mitigated as well.
| Reporter | ||
Comment 11•5 months ago
|
||
A Pernosco session is available here: https://pernos.co/debug/qhnXNZ5nfJmPaDJhsvRm4g/index.html
Let me know if this recording is not sufficient and we can try something else.
Updated•5 months ago
|
| Assignee | ||
Comment 13•5 months ago
•
|
||
[apologies for typos; it's late and I'm a bit bleary-eyed]
(In reply to Tyson Smith [:tsmith] from comment #11)
A Pernosco session is available here: https://pernos.co/debug/qhnXNZ5nfJmPaDJhsvRm4g/index.html
Let me know if this recording is not sufficient and we can try something else.
I just took a look. Unfortunately this recording doesn't seem to have debug symbols; it can't show the value of variables in the "Local Variables" pane, and gdb can't print variables by name either. It does look like we've got register values, which makes it somewhat actionable. I don't have a ton of experience doing gdb/pernosco debugging with register values, but I think I managed to understand what was going on (up until I ran into a mystery, see below).
As best as I can tell, here's what's going on when TSAN complains there:
- we have an entirely valid frame and property-table, which only contains a single entry, with descriptor
IBSplitPrevSibling()(notePrevin the middle of the name there, which makes it not the descriptor we end up looking for) - we get a call to
blockFrame->GetProperty(nsIFrame::IBSplitSibling());which should be doomed to fail (i.e. return nullptr), because we don't have any IBSplitSibling-flavored property table entry. [1] But it doesn't fail in the way that I would expect it to! - Inside that
GetPropertycall, we search the property table by walking a nsTArray, applying a comparison function on each array-entry, to check for whether it'snsIFrame::IBSplitSibling()::descriptor-- that happens here (via a callsite inFrameProperties::GetInternalwhich passes in two comparator functions):
auto ApplyIf(const Item& aItem, index_type aStart, const Comparator& aComp,
Function&& aFunction, FunctionElse&& aFunctionElse) const {
...
const value_type* const elements = Elements();
const value_type* const iend = elements + Length();
for (const value_type* iter = elements + aStart; iter != iend; ++iter) {
if (comp.Equals(*iter, aItem)) {
return InvokeWithIndexAndOrReference<const value_type>(
std::forward<Function>(aFunction), iter - elements, *iter);
}
}
return aFunctionElse();
Given that this array has a length of 1 (which I verified from inspecting its Hdr()->mLength), this for-loop should only execute once. BUT, in this pernosco trace, we execute the loop at least 3 times, advancing iter through the nsTArray's "Elements()" buffer and calling the comparison function on each successive spot in memory. And as soon as we read off the end of the Elements() buffer (which happens quickly), the TSAN instrumentation naturally complains.
And the weird thing is, I'm not seeing any instructions that look like they would be the iter != iend loop-condition check from the code-quote above (comparing the pointer to the current Elements() entry against some sentinel value that would represent iend, or an index-vs-length comparison, etc). Perhaps the compiler generated a loop-unrolled version of this function, and it assumed we were going to run this loop at least 3 times for some reason? and hence there aren't any iter != iend checks that happen before the point where we crash.
I don't know why that would happen offhand. I'm tempted to suspect some weird compiler optimization that's misfiring and optimizing away the iter != iend; comparison when we compile with tsan, for some reason.... I don't see any other clear explanation from the pernosco trace right now. If we were actually reading off the end of the array in real builds (as we do in pernosco), then ASAN would be able to pick that up as well.
[1] this should result in a null-deref, and that's the way in which I would expect our blockFrame->GetProperty(nsIFrame::IBSplitSibling()); call to be doomed. That's a legit bug, but it's just a null-deref -- it's what dshin described in comment 6.
| Assignee | ||
Comment 14•5 months ago
|
||
So: bottom line, we've got a real null-deref bug here (comment 6), but I'm suspicious that the TSAN issue is a compiler bug.
| Assignee | ||
Comment 15•5 months ago
|
||
Given that analysis, I'm downgrading this to S3 for now.
| Assignee | ||
Comment 16•5 months ago
•
|
||
(and we can probably consider this sec-moderate rather than sec-high. But if we see evidence of this reproducing in ASAN builds (or really non-TSAN builds), then I'd get worried again.)
| Assignee | ||
Updated•5 months ago
|
Comment 17•5 months ago
|
||
It seems like the fuzzer found the null deref - bug 2031808.
| Assignee | ||
Comment 18•5 months ago
|
||
Hmm, bug 2031808's fuzzer testcase causes stock Nightly builds to crash, with a non-null pointer address, on the same line of code that TSAN is warning us about here (inside mozilla::FrameProperties::PropertyComparator::Equals). So that lends credibility to this being a real Gecko bug rather than some TSAN/compiler quirk as I was suspecting...
Let's restore sec-high/csectype-uaf/s2, and I'll keep poking here.
| Assignee | ||
Comment 19•4 months ago
•
|
||
OK, I've sorted this out, and tl;dr this is sort of a nullptr deref, but it's worse because the compiler is applying a fancy optimization that turns it into an out-of-bounds read. But the good news is that a null-check is the correct fix.
Details below, elaborating on this observation from comment 13:
And the weird thing is, I'm not seeing any instructions that look like they would be the
iter != iendloop-condition check
I confirmed that there is indeed no code generated for the loop-condition-check at all in the disassembly for UpdateStyleOfOwnedAnonBoxesForIBSplit (which has GetProperty fully inlined inside of it, including the loop over the property table's elements).
That lack-of-a-loop-condition is surprising, but it makes sense given aggressive compiler optimization.
Here's what's happening:
- The compiler sees that in nsTArray.h
ApplyIf(), the code after the loop returns nullptr (which is abstracted asaFunctionElse). - The compiler also sees that we fail to check the returned value for nullptr.
- So it assumes that the code beyond the loop is never reached -- i.e. it assumes that the loop condition will never fail and it optimizes away the check for the loop condition.
In other words, the compiler converts a for loop like the following (with a nice bounds-check)...
for (const value_type* iter = elements + aStart; iter != iend; ++iter) {
// code that dereferences 'iter' and returns if it finds what it's looking for
}
return nullptr;
...to an infinite while loop with no bounds check:
const value_type* iter = elements + aStart;
while (true) {
// code that dereferences 'iter' and returns if it finds what it's looking for
++iter;
}
And the latter formulation will happily read beyond the end of the elements array if it doesn't find the sort of match that it's looking for.
| Assignee | ||
Updated•4 months ago
|
| Assignee | ||
Updated•4 months ago
|
| Assignee | ||
Updated•4 months ago
|
| Assignee | ||
Comment 21•4 months ago
|
||
Comment 22•4 months ago
|
||
Copying crash signatures from duplicate bugs.
| Assignee | ||
Updated•4 months ago
|
| Assignee | ||
Comment 23•4 months ago
|
||
Comment on attachment 9571319 [details]
(secure)
Security Approval Request
- How easily could an exploit be constructed based on the patch?: Not easily.
- Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
- Which branches (beta, release, and/or ESR) are affected by this flaw, and do the release status flags reflect this affected/unaffected state correctly?: All
- If not all supported branches, which bug introduced the flaw?: None
- Do you have backports for the affected branches?: No
- If not, how different, hard to create, and risky will they be?: Trivial
- How likely is this patch to cause regressions; how much testing does it need?: Very unlikely. It's just adding a null-check to gracefully handle a condition that would otherwise result in a crash.
- Is the patch ready to land after security approval is given?: Yes
- Is Android affected?: Yes
Updated•4 months ago
|
Updated•4 months ago
|
Updated•4 months ago
|
| Assignee | ||
Comment 24•4 months ago
|
||
Original Revision: https://phabricator.services.mozilla.com/D295119
Updated•4 months ago
|
Comment 25•4 months ago
|
||
firefox-beta Uplift Approval Request
- User impact if declined/Reason for urgency: Security vulnerability - potential for UAF (reading past the end of a nsTArray's internal buffer).
- Code covered by automated testing?: no
- Fix verified in Nightly?: no
- Needs manual QE testing?: yes
- Steps to reproduce for manual QE testing: Make several attempts to load/reload this attachment several times (the testcase on our duplicate bug 2031808, which I've found is the easiest way to trigger this):
https://bugzilla.mozilla.org/attachment.cgi?id=9569942 - Risk associated with taking this patch: low
- Explanation of risk level: This is just adding a null-check to avoid a crash. (This bug is essentially a case where we're missing a null-check to cover an unusual case, though we end up crashing with a worrisome non-null address for reasons discussed in comment 19.)
- String changes made/needed?: None
- Is Android affected?: yes
| Assignee | ||
Comment 26•4 months ago
|
||
Original Revision: https://phabricator.services.mozilla.com/D295119
Updated•4 months ago
|
Comment 27•4 months ago
|
||
firefox-esr140 Uplift Approval Request
- User impact if declined/Reason for urgency: Security vulnerability - potential for UAF (reading past the end of a nsTArray's internal buffer).
- Code covered by automated testing?: no
- Fix verified in Nightly?: no
- Needs manual QE testing?: yes
- Steps to reproduce for manual QE testing: Make several attempts to load/reload this attachment several times (the testcase on our duplicate bug 2031808, which I've found is the easiest way to trigger this):
https://bugzilla.mozilla.org/attachment.cgi?id=9569942 - Risk associated with taking this patch: low
- Explanation of risk level: This is just adding a null-check to avoid a crash. (This bug is essentially a case where we're missing a null-check to cover an unusual case, though we end up crashing with a worrisome non-null address for reasons discussed in comment 19.)
- String changes made/needed?: None
- Is Android affected?: yes
| Assignee | ||
Comment 28•4 months ago
|
||
Original Revision: https://phabricator.services.mozilla.com/D295119
Updated•4 months ago
|
Comment 29•4 months ago
|
||
firefox-release Uplift Approval Request
- User impact if declined/Reason for urgency: Security vulnerability - potential for UAF (reading past the end of a nsTArray's internal buffer).
- Code covered by automated testing?: no
- Fix verified in Nightly?: no
- Needs manual QE testing?: yes
- Steps to reproduce for manual QE testing: Make several attempts to load/reload this attachment several times (the testcase on our duplicate bug 2031808, which I've found is the easiest way to trigger this):
https://bugzilla.mozilla.org/attachment.cgi?id=9569942 - Risk associated with taking this patch: low
- Explanation of risk level: This is just adding a null-check to avoid a crash. (This bug is essentially a case where we're missing a null-check to cover an unusual case, though we end up crashing with a worrisome non-null address for reasons discussed in comment 19.)
- String changes made/needed?: None
- Is Android affected?: yes
Comment 30•4 months ago
|
||
Comment 31•4 months ago
|
||
Updated•4 months ago
|
Updated•4 months ago
|
Comment 32•4 months ago
|
||
| uplift | ||
Comment 33•4 months ago
|
||
Verified bug as fixed on rev mozilla-central 20260421085657-a2d0bb819f99.
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.
Updated•4 months ago
|
Updated•4 months ago
|
Comment 34•4 months ago
|
||
| uplift | ||
| Assignee | ||
Comment 35•4 months ago
•
|
||
Good news backport/uplift-wise -- esr115 actually appears to be unaffected here, because it had a null-check here, which avoided this bug (it's the same null-check that this bug's patch is adding back):
https://searchfox.org/firefox-esr115/rev/d76bc01a7004a86bf9966b4cf736655aa1499cc5/layout/generic/nsInlineFrame.cpp#908-913
That check was removed in bug 1931332 (in Firefox 134).
So I guess technically this bug here is a regression from bug 1931332, and the fix here is backing out bug 1931332 in a sense.
--> Marking as-such, for version tracking purposes at least. (And marking 115 as unaffected.)
Updated•4 months ago
|
Updated•4 months ago
|
Comment 36•4 months ago
|
||
Repro the crash with test case from https://bugzilla.mozilla.org/show_bug.cgi?id=2031808#c0, using an affected Nightly ASan build, on Ubuntu 26.04.
The issue is verified as fixed on latest ASan Esr 140.11.0 and Beta 151.0b2 under Ubuntu 26.04.
Updated•4 months ago
|
Comment 37•4 months ago
|
||
| uplift | ||
Updated•4 months ago
|
Updated•4 months ago
|
Comment 38•4 months ago
|
||
| 140.10.1 uplift | ||
Updated•4 months ago
|
Updated•4 months ago
|
Updated•4 months ago
|
Updated•4 months ago
|
Comment 39•4 months ago
|
||
This is also verified as fixed on Firefox 150.0.1 ASan build running Ubuntu 26.04.
Updated•16 days ago
|
Description
•