1431885 - UBSan: gfx/src/nsCoord.h:100:18: runtime error: -nan is outside the range of representable values of type 'int'

Status: NEW

(Core :: Layout, defect, P3)

Description

[Tyson Smith [:tsmith]]

This error is triggered when Preferences->Search page is opened on a Firefox build built with: -fsanitize=float-cast-overflow

changeset: 399986:6bb6f3b25f9f

gfx/src/nsCoord.h:100:18: runtime error: -nan is outside the range of representable values of type 'int'
    #0 0x7f17dce5b5fc in NSToCoordRound(double) gfx/src/nsCoord.h:100:18
    #1 0x7f17e20ac5e9 in nsLayoutUtils::GetWholeImageDestination(nsSize const&, nsRect const&, nsRect const&) layout/base/nsLayoutUtils.cpp:7308:25
    #2 0x7f17e2674c87 in nsTreeBodyFrame::PaintImage(int, nsTreeColumn*, nsRect const&, nsPresContext*, gfxContext&, nsRect const&, int&, int&, nsDisplayListBuilder*) layout/xul/tree/nsTreeBodyFrame.cpp:3649:9
    #3 0x7f17e26728f5 in nsTreeBodyFrame::PaintCell(int, nsTreeColumn*, nsRect const&, nsPresContext*, gfxContext&, nsRect const&, int&, nsPoint, nsDisplayListBuilder*) layout/xul/tree/nsTreeBodyFrame.cpp:3384:15
    #4 0x7f17e266f8cd in nsTreeBodyFrame::PaintRow(int, nsRect const&, nsPresContext*, gfxContext&, nsRect const&, nsPoint, nsDisplayListBuilder*) layout/xul/tree/nsTreeBodyFrame.cpp:3160:21
    #5 0x7f17e266dbaa in nsTreeBodyFrame::PaintTreeBody(gfxContext&, nsRect const&, nsPoint, nsDisplayListBuilder*) layout/xul/tree/nsTreeBodyFrame.cpp:2953:17
    #6 0x7f17e2698f67 in nsDisplayTreeBody::Paint(nsDisplayListBuilder*, gfxContext*) layout/xul/tree/nsTreeBodyFrame.cpp:2814:9
    #7 0x7f17e276ffc8 in mozilla::FrameLayerBuilder::PaintItems(nsTArray<mozilla::FrameLayerBuilder::ClippedDisplayItem>&, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const&, gfxContext*, nsDisplayListBuilder*, nsPresContext*, mozilla::gfx::IntPointTyped<mozilla::gfx::UnknownUnits> const&, float, float, int) layout/painting/FrameLayerBuilder.cpp:6029:21
    #8 0x7f17e2770fe7 in mozilla::FrameLayerBuilder::DrawPaintedLayer(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*) layout/painting/FrameLayerBuilder.cpp:6190:19
    #9 0x7f17dd3b22d4 in mozilla::layers::ClientPaintedLayer::PaintThebes(nsTArray<mozilla::layers::ReadbackProcessor::Update>*) gfx/layers/client/ClientPaintedLayer.cpp:158:5
    #10 0x7f17dd3b3d82 in mozilla::layers::ClientPaintedLayer::RenderLayerWithReadback(mozilla::layers::ReadbackProcessor*) gfx/layers/client/ClientPaintedLayer.cpp:314:3
    #11 0x7f17dd3f9bf0 in mozilla::layers::ClientContainerLayer::RenderLayer() gfx/layers/client/ClientContainerLayer.h:58:29
    #12 0x7f17dd3f9bf0 in mozilla::layers::ClientContainerLayer::RenderLayer() gfx/layers/client/ClientContainerLayer.h:58:29
    #13 0x7f17dd3ad243 in mozilla::layers::ClientLayerManager::EndTransactionInternal(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::LayerManager::EndTransactionFlags) gfx/layers/client/ClientLayerManager.cpp:359:13
    #14 0x7f17dd3adb03 in mozilla::layers::ClientLayerManager::EndTransaction(void (*)(mozilla::layers::PaintedLayer*, gfxContext*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::layers::DrawRegionClip, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, void*), void*, mozilla::layers::LayerManager::EndTransactionFlags) gfx/layers/client/ClientLayerManager.cpp:423:3
    #15 0x7f17e27d6209 in nsDisplayList::PaintRoot(nsDisplayListBuilder*, gfxContext*, unsigned int) layout/painting/nsDisplayList.cpp:2623:17
    #16 0x7f17e209c141 in nsLayoutUtils::PaintFrame(gfxContext*, nsIFrame*, nsRegion const&, unsigned int, nsDisplayListBuilderMode, nsLayoutUtils::PaintFrameFlags) layout/base/nsLayoutUtils.cpp:3976:12
    #17 0x7f17e1fbb12a in mozilla::PresShell::Paint(nsView*, nsRegion const&, unsigned int) layout/base/PresShell.cpp:6481:5
    #18 0x7f17e186db3b in nsViewManager::ProcessPendingUpdatesPaint(nsIWidget*) view/nsViewManager.cpp:480:19
    #19 0x7f17e186d21d in nsViewManager::ProcessPendingUpdatesForView(nsView*, bool) view/nsViewManager.cpp:412:33
    #20 0x7f17e186f5cb in nsViewManager::ProcessPendingUpdates() view/nsViewManager.cpp:1102:5
    #21 0x7f17e1f3684c in nsRefreshDriver::Tick(long, mozilla::TimeStamp) layout/base/nsRefreshDriver.cpp:2046:11
    #22 0x7f17e1f39ac8 in nsRefreshDriver::FinishedWaitingForTransaction() layout/base/nsRefreshDriver.cpp:2154:5
    #23 0x7f17dd3afc2c in mozilla::layers::ClientLayerManager::DidComposite(unsigned long, mozilla::TimeStamp const&, mozilla::TimeStamp const&) gfx/layers/client/ClientLayerManager.cpp:523:32
    #24 0x7f17dd4bbf89 in mozilla::layers::CompositorBridgeChild::RecvDidComposite(unsigned long const&, unsigned long const&, mozilla::TimeStamp const&, mozilla::TimeStamp const&) gfx/layers/ipc/CompositorBridgeChild.cpp:543:8
    #25 0x7f17dc1f6a9b in mozilla::layers::PCompositorBridgeChild::OnMessageReceived(IPC::Message const&) objdir-ff-ubsan/ipc/ipdl/PCompositorBridgeChild.cpp:1441:20
    #26 0x7f17db6cd4bf in mozilla::ipc::MessageChannel::DispatchAsyncMessage(IPC::Message const&) ipc/glue/MessageChannel.cpp:2110:25
    #27 0x7f17db6cb577 in mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) ipc/glue/MessageChannel.cpp:2040:17
    #28 0x7f17db6cc034 in mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::MessageChannel::MessageTask&) ipc/glue/MessageChannel.cpp:1886:5
    #29 0x7f17db6cc723 in mozilla::ipc::MessageChannel::MessageTask::Run() ipc/glue/MessageChannel.cpp:1919:15
    #30 0x7f17da413d9d in nsThread::ProcessNextEvent(bool, bool*) xpcom/threads/nsThread.cpp:1040:14
    #31 0x7f17da44ec0a in NS_ProcessNextEvent(nsIThread*, bool) xpcom/threads/nsThreadUtils.cpp:517:10
    #32 0x7f17db6d3f31 in mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) ipc/glue/MessagePump.cpp:97:21
    #33 0x7f17db53e910 in MessageLoop::Run() ipc/chromium/src/base/message_loop.cc:299:3
    #34 0x7f17e19033f5 in nsBaseAppShell::Run() widget/nsBaseAppShell.cpp:157:27
    #35 0x7f17e72c2047 in nsAppStartup::Run() toolkit/components/startup/nsAppStartup.cpp:288:30
    #36 0x7f17e7495f88 in XREMain::XRE_mainRun() toolkit/xre/nsAppRunner.cpp:4702:22
    #37 0x7f17e7497e3f in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) toolkit/xre/nsAppRunner.cpp:4841:8
    #38 0x7f17e7498cb1 in XRE_main(int, char**, mozilla::BootstrapConfig const&) toolkit/xre/nsAppRunner.cpp:4933:21
    #39 0x51851e in do_main(int, char**, char**) browser/app/nsBrowserApp.cpp:231:22
    #40 0x517d24 in main browser/app/nsBrowserApp.cpp:304:16
    #41 0x7f1811a391c0 in __libc_start_main /build/glibc-itYbWN/glibc-2.26/csu/../csu/libc-start.c:308
    #42 0x4207a9 in _start (objdir-ff-ubsan/dist/bin/firefox+0x4207a9)

Comment 1

[Xidorn Quan [:xidorn] UTC+10]

https://searchfox.org/mozilla-central/rev/4611b9541894e90a421debb57ddbbcff55c2f369/gfx/src/nsCoord.h#100
https://searchfox.org/mozilla-central/rev/4611b9541894e90a421debb57ddbbcff55c2f369/layout/base/nsLayoutUtils.cpp#7308

The code is:
> nscoord destOffsetX = NSToCoordRound(aImageSourceArea.x*scaleX);
in which aImageSourceArea is an nsRect so aImageSourceArea.x is an integer, and scaleX is from
> double scaleX = double(aDestArea.width)/aImageSourceArea.width;
where all the involving parts are integers.

Two integers division producing nan sounds like a 0/0 case. The callsite of this function is
https://searchfox.org/mozilla-central/rev/4611b9541894e90a421debb57ddbbcff55c2f369/layout/xul/tree/nsTreeBodyFrame.cpp#3649

Looks like both rawImageSize and sourceRect in the function have a zero width. I have no idea how to handle this case, then. But it sounds like an issue in XUL layout...

Comment 2

[Jet Villegas (inactive)]

[ Triage 2017/02/20: P3 ]

Comment 3

[Tyson Smith [:tsmith]]

Attachment: testcase.html