1759021 - src/layout/painting/nsDisplayList.cpp:6472:39: runtime error: 2.14748e+09 is outside the range of representable values of type 'int'

Status: NEW

(Core :: Web Painting, defect, P3)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.html

This was found by enabling the float-cast-overflow check in UBSan and fuzzing. This type of issue can create inconsistencies across platforms, architectures and optimization levels.

Found with m-c 20220309-ae667f73a8f1

To enable this check add the following to your mozconfig:

ac_add_options --enable-undefined-sanitizer="float-cast-overflow"

src/layout/painting/nsDisplayList.cpp:6472:39: runtime error: 2.14748e+09 is outside the range of representable values of type 'int'
    #0 0x7f1ac5d3f098 in mozilla::nsDisplayTransform::ShouldPrerenderTransformedContent(mozilla::nsDisplayListBuilder*, nsIFrame*, nsRect*) src/layout/painting/nsDisplayList.cpp:6472:39
    #1 0x7f1ac585af0b in nsIFrame::BuildDisplayListForStackingContext(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayList*, bool*) src/layout/generic/nsIFrame.cpp:3263:21
    #2 0x7f1ac5781460 in nsIFrame::BuildDisplayListForChild(mozilla::nsDisplayListBuilder*, nsIFrame*, mozilla::nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) src/layout/generic/nsIFrame.cpp:4355:12
    #3 0x7f1ac577fbae in nsFlexContainerFrame::BuildDisplayList(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayListSet const&) src/layout/generic/nsFlexContainerFrame.cpp:2940:5
    #4 0x7f1ac585bd63 in nsIFrame::BuildDisplayListForStackingContext(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayList*, bool*) src/layout/generic/nsIFrame.cpp:3504:5
    #5 0x7f1ac5781460 in nsIFrame::BuildDisplayListForChild(mozilla::nsDisplayListBuilder*, nsIFrame*, mozilla::nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) src/layout/generic/nsIFrame.cpp:4355:12
    #6 0x7f1ac577fbae in nsFlexContainerFrame::BuildDisplayList(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayListSet const&) src/layout/generic/nsFlexContainerFrame.cpp:2940:5
    #7 0x7f1ac585bd63 in nsIFrame::BuildDisplayListForStackingContext(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayList*, bool*) src/layout/generic/nsIFrame.cpp:3504:5
    #8 0x7f1ac5781460 in nsIFrame::BuildDisplayListForChild(mozilla::nsDisplayListBuilder*, nsIFrame*, mozilla::nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) src/layout/generic/nsIFrame.cpp:4355:12
    #9 0x7f1ac577fbae in nsFlexContainerFrame::BuildDisplayList(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayListSet const&) src/layout/generic/nsFlexContainerFrame.cpp:2940:5
    #10 0x7f1ac585bd63 in nsIFrame::BuildDisplayListForStackingContext(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayList*, bool*) src/layout/generic/nsIFrame.cpp:3504:5
    #11 0x7f1ac5781460 in nsIFrame::BuildDisplayListForChild(mozilla::nsDisplayListBuilder*, nsIFrame*, mozilla::nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) src/layout/generic/nsIFrame.cpp:4355:12
    #12 0x7f1ac5744125 in nsCanvasFrame::BuildDisplayList(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayListSet const&) src/layout/generic/nsCanvasFrame.cpp:614:5
    #13 0x7f1ac5863404 in nsIFrame::BuildDisplayListForSimpleChild(mozilla::nsDisplayListBuilder*, nsIFrame*, mozilla::nsDisplayListSet const&) src/layout/generic/nsIFrame.cpp:4067:11
    #14 0x7f1ac5780a53 in nsIFrame::BuildDisplayListForChild(mozilla::nsDisplayListBuilder*, nsIFrame*, mozilla::nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) src/layout/generic/nsIFrame.cpp:4180:5
    #15 0x7f1ac57f5349 in mozilla::ScrollFrameHelper::BuildDisplayList(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayListSet const&) src/layout/generic/nsGfxScrollFrame.cpp:4044:15
    #16 0x7f1ac58d952f in nsHTMLScrollFrame::BuildDisplayList(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayListSet const&) src/layout/generic/nsGfxScrollFrame.h:866:13
    #17 0x7f1ac5863404 in nsIFrame::BuildDisplayListForSimpleChild(mozilla::nsDisplayListBuilder*, nsIFrame*, mozilla::nsDisplayListSet const&) src/layout/generic/nsIFrame.cpp:4067:11
    #18 0x7f1ac5780a53 in nsIFrame::BuildDisplayListForChild(mozilla::nsDisplayListBuilder*, nsIFrame*, mozilla::nsDisplayListSet const&, mozilla::EnumSet<nsIFrame::DisplayChildFlag, unsigned int>) src/layout/generic/nsIFrame.cpp:4180:5
    #19 0x7f1ac5707ff1 in mozilla::ViewportFrame::BuildDisplayList(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayListSet const&) src/layout/generic/ViewportFrame.cpp:66:3
    #20 0x7f1ac585bd63 in nsIFrame::BuildDisplayListForStackingContext(mozilla::nsDisplayListBuilder*, mozilla::nsDisplayList*, bool*) src/layout/generic/nsIFrame.cpp:3504:5
    #21 0x7f1ac5cb23ac in mozilla::RetainedDisplayListBuilder::AttemptPartialUpdate(unsigned int) src/layout/painting/RetainedDisplayListBuilder.cpp:1699:25
    #22 0x7f1ac5631615 in nsLayoutUtils::PaintFrame(gfxContext*, nsIFrame*, nsRegion const&, unsigned int, mozilla::nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) src/layout/base/nsLayoutUtils.cpp:3337:40
    #23 0x7f1ac555a072 in mozilla::PresShell::PaintInternal(nsView*, mozilla::PaintInternalFlags) src/layout/base/PresShell.cpp:6362:5
    #24 0x7f1ac5559470 in mozilla::PresShell::PaintAndRequestComposite(nsView*, mozilla::PaintFlags) src/layout/base/PresShell.cpp:6233:3
    #25 0x7f1ac4ebdf15 in nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*) src/view/nsViewManager.cpp:440:18
    #26 0x7f1ac4ebd756 in nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) src/view/nsViewManager.cpp:375:22
    #27 0x7f1ac4ebf1bb in nsViewManager::ProcessPendingUpdates() src/view/nsViewManager.cpp:948:5
    #28 0x7f1ac54d8eb3 in nsRefreshDriver::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsRefreshDriver::IsExtraTick) src/layout/base/nsRefreshDriver.cpp:2580:11
    #29 0x7f1ac54e8bb3 in mozilla::RefreshDriverTimer::TickDriver(nsRefreshDriver*, mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) src/layout/base/nsRefreshDriver.cpp:350:13
    #30 0x7f1ac54e8bb3 in mozilla::RefreshDriverTimer::TickRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp, nsTArray<RefPtr<nsRefreshDriver> >&) src/layout/base/nsRefreshDriver.cpp:328:7
    #31 0x7f1ac54e8845 in mozilla::RefreshDriverTimer::Tick(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) src/layout/base/nsRefreshDriver.cpp:344:5
    #32 0x7f1ac54e8337 in mozilla::VsyncRefreshDriverTimer::RunRefreshDrivers(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) src/layout/base/nsRefreshDriver.cpp:788:5
    #33 0x7f1ac54e7969 in mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::TickRefreshDriver(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::TimeStamp) src/layout/base/nsRefreshDriver.cpp:693:16
    #34 0x7f1ac54e6b58 in mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyVsyncOnMainThread() src/layout/base/nsRefreshDriver.cpp:610:7
    #35 0x7f1ac54e64e9 in mozilla::VsyncRefreshDriverTimer::RefreshDriverVsyncObserver::NotifyVsync(mozilla::VsyncEvent const&) src/layout/base/nsRefreshDriver.cpp:516:9
    #36 0x7f1ac425087a in mozilla::dom::VsyncMainChild::RecvNotify(mozilla::VsyncEvent const&, float const&) src/dom/ipc/VsyncMainChild.cpp:68:15
    #37 0x7f1ac4653f6a in mozilla::dom::PVsyncChild::OnMessageReceived(IPC::Message const&) src/objdir-ff-ubsan/ipc/ipdl/PVsyncChild.cpp:220:54
    #38 0x7f1abe498626 in mozilla::ipc::PBackgroundChild::OnMessageReceived(IPC::Message const&) src/objdir-ff-ubsan/ipc/ipdl/PBackgroundChild.cpp:6370:32
    #39 0x7f1abe3f8a48 in mozilla::ipc::MessageChannel::DispatchAsyncMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message const&) src/ipc/glue/MessageChannel.cpp:1674:25
    #40 0x7f1abe3f6436 in mozilla::ipc::MessageChannel::DispatchMessage(mozilla::ipc::ActorLifecycleProxy*, IPC::Message&&) src/ipc/glue/MessageChannel.cpp:1599:9
    #41 0x7f1abe3f6e66 in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::ActorLifecycleProxy*, mozilla::ipc::MessageChannel::MessageTask&) src/ipc/glue/MessageChannel.cpp:1462:3
    #42 0x7f1abe3f7885 in mozilla::ipc::MessageChannel::MessageTask::Run() src/ipc/glue/MessageChannel.cpp:1496:14
    #43 0x7f1abcd4dafa in mozilla::RunnableTask::Run() src/xpcom/threads/TaskController.cpp:467:16
    #44 0x7f1abcd0e25f in mozilla::TaskController::DoExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) src/xpcom/threads/TaskController.cpp:770:26
    #45 0x7f1abcd0b8ae in mozilla::TaskController::ExecuteNextTaskOnlyMainThreadInternal(mozilla::detail::BaseAutoLock<mozilla::Mutex&> const&) src/xpcom/threads/TaskController.cpp:606:15
    #46 0x7f1abcd0c004 in mozilla::TaskController::ProcessPendingMTTask(bool) src/xpcom/threads/TaskController.cpp:390:36
    #47 0x7f1abcd3f2f1 in mozilla::TaskController::InitializeInternal()::$_0::operator()() const src/xpcom/threads/TaskController.cpp:124:37
    #48 0x7f1abcd3f2f1 in mozilla::detail::RunnableFunction<mozilla::TaskController::InitializeInternal()::$_0>::Run() src/objdir-ff-ubsan/dist/include/nsThreadUtils.h:531:5
    #49 0x7f1abcd2a433 in nsThread::ProcessNextEvent(bool, bool*) src/xpcom/threads/nsThread.cpp:1173:16
    #50 0x7f1abcd33af4 in NS_ProcessNextEvent(nsIThread*, bool) src/xpcom/threads/nsThreadUtils.cpp:467:10
    #51 0x7f1abe3ff902 in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:85:21
    #52 0x7f1abe400ee2 in mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:268:30
    #53 0x7f1abe26fb11 in MessageLoop::RunInternal() src/ipc/chromium/src/base/message_loop.cc:331:10
    #54 0x7f1abe26fb11 in MessageLoop::RunHandler() src/ipc/chromium/src/base/message_loop.cc:324:3
    #55 0x7f1abe26fb11 in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:306:3
    #56 0x7f1ac4fb5e08 in nsBaseAppShell::Run() src/widget/nsBaseAppShell.cpp:137:27
    #57 0x7f1ac9e34d27 in XRE_RunAppShell() src/toolkit/xre/nsEmbedFunctions.cpp:870:20
    #58 0x7f1abe400ec1 in mozilla::ipc::MessagePumpForChildProcess::Run(base::MessagePump::Delegate*) src/ipc/glue/MessagePump.cpp:235:9
    #59 0x7f1abe26fb11 in MessageLoop::RunInternal() src/ipc/chromium/src/base/message_loop.cc:331:10
    #60 0x7f1abe26fb11 in MessageLoop::RunHandler() src/ipc/chromium/src/base/message_loop.cc:324:3
    #61 0x7f1abe26fb11 in MessageLoop::Run() src/ipc/chromium/src/base/message_loop.cc:306:3
    #62 0x7f1ac9e33e7d in XRE_InitChildProcess(int, char**, XREChildData const*) src/toolkit/xre/nsEmbedFunctions.cpp:729:34
    #63 0x7f1ac9e49020 in mozilla::BootstrapImpl::XRE_InitChildProcess(int, char**, XREChildData const*) src/toolkit/xre/Bootstrap.cpp:67:12
    #64 0x55c732124495 in content_process_main(mozilla::Bootstrap*, int, char**) src/browser/app/../../ipc/contentproc/plugin-container.cpp:57:28
    #65 0x55c7321248a5 in main src/browser/app/nsBrowserApp.cpp:327:18
    #66 0x7f1ae5d21c86 in __libc_start_main /build/glibc-uZu3wS/glibc-2.27/csu/../csu/libc-start.c:310
    #67 0x55c732073578 in _start (src/objdir-ff-ubsan/dist/bin/firefox+0xf4578)

Comment 1

[Tyson Smith [:tsmith]]

Please ni? me if a Pernosco session would be helpful.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:miko, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:miko, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 4

[Tyson Smith [:tsmith]]

This issue is currently triggered while fuzzing with the 'float-cast-overflow' UBSan check enabled. This issue will need to be addressed before the check can be enabled by default.

If it requires too much effort to fix immediately please ni? me and let me know. If necessary it will be added to a suppression list. Thank you :)

Comment 5

[Miko Mynttinen]

I could not reproduce this locally. Does the testcase require special prefs?

Comment 6

[Tyson Smith [:tsmith]]

Attachment: prefs.js

Maybe, here is what the fuzzer would have used.

You could also try Grizzly Replay which will generate the prefs and setup the environment.

$ pip install grizzly-framework
$ python -m grizzly.replay <firefox_build> testcase.html --xvfb

Comment 7

[Tyson Smith [:tsmith]]

I can still reproduce with m-c 20220608-fab53b92a95d. I tried both -O1 and -O2.

Comment 8

[Tyson Smith [:tsmith]]

Miko, would a Pernosco session be helpful?

Comment 9

[Miko Mynttinen]

I will not be able to work on this.