1663711 - Assertion failure: aTransform.PreservesAxisAlignedRectangles(), at /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:288

Status: RESOLVED WONTFIX

(Core :: Graphics: Layers, defect)

Description

[Jason Kratzer [:jkratzer]]

Attachment: testcase.html

Testcase found while fuzzing mozilla-central rev fb9c01b719fa (built with --enable-debug).

Assertion failure: aTransform.PreservesAxisAlignedRectangles(), at /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:288

    #0 0x7fa7aa331ab3 in TransformRect<mozilla::gfx::UnknownUnits> /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:288:5
    #1 0x7fa7aa331ab3 in TransformRectRoundIn<mozilla::gfx::UnknownUnits, mozilla::gfx::BaseMatrix<float> > /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:329:10
    #2 0x7fa7aa331ab3 in void mozilla::layers::AddTransformedRegionRoundIn<mozilla::gfx::UnknownUnits, mozilla::gfx::BaseMatrix<float> >(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits> const&, mozilla::gfx::BaseMatrix<float> const&) /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:347:21
    #3 0x7fa7aa330b73 in mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTargetPixel> > const&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::ParentLayerPixel> > const&, bool) /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:474:5
    #4 0x7fa7aa3309cf in mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTargetPixel> > const&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::ParentLayerPixel> > const&, bool) /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:450:7
    #5 0x7fa7aa3309cf in mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTargetPixel> > const&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::ParentLayerPixel> > const&, bool) /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:450:7
    #6 0x7fa7aa3309cf in mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTargetPixel> > const&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::ParentLayerPixel> > const&, bool) /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:450:7
    #7 0x7fa7aa3309cf in mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTargetPixel> > const&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::ParentLayerPixel> > const&, bool) /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:450:7
    #8 0x7fa7aa32fbef in mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&) /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:355:3
    #9 0x7fa7aa331fbb in mozilla::layers::LayerManagerComposite::UpdateAndRender() /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:604:3
    #10 0x7fa7aa331d14 in mozilla::layers::LayerManagerComposite::EndTransaction(mozilla::TimeStamp const&, mozilla::layers::LayerManager::EndTransactionFlags) /builds/worker/checkouts/gecko/gfx/layers/composite/LayerManagerComposite.cpp:579:5
    #11 0x7fa7aa37638b in mozilla::layers::CompositorBridgeParent::CompositeToTarget(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::gfx::DrawTarget*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*) /builds/worker/checkouts/gecko/gfx/layers/ipc/CompositorBridgeParent.cpp:1042:18
    #12 0x7fa7aa38cf30 in mozilla::layers::CompositorVsyncScheduler::Composite(mozilla::VsyncEvent const&) /builds/worker/checkouts/gecko/gfx/layers/ipc/CompositorVsyncScheduler.cpp:256:27
    #13 0x7fa7aa3af738 in applyImpl<mozilla::layers::CompositorVsyncScheduler, void (mozilla::layers::CompositorVsyncScheduler::*)(const mozilla::VsyncEvent &), StoreCopyPassByConstLRef<mozilla::VsyncEvent> , 0> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1188:12
    #14 0x7fa7aa3af738 in apply<mozilla::layers::CompositorVsyncScheduler, void (mozilla::layers::CompositorVsyncScheduler::*)(const mozilla::VsyncEvent &)> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1194:12
    #15 0x7fa7aa3af738 in mozilla::detail::RunnableMethodImpl<mozilla::layers::CompositorVsyncScheduler*, void (mozilla::layers::CompositorVsyncScheduler::*)(mozilla::VsyncEvent const&), true, (mozilla::RunnableKind)1, mozilla::VsyncEvent>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1240:13
    #16 0x7fa7a8abc47f in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1234:14
    #17 0x7fa7a8ac1e2a in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:513:10
    #18 0x7fa7a93baf16 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:332:5
    #19 0x7fa7a932c9e3 in MessageLoop::RunInternal() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:334:10
    #20 0x7fa7a932c8fd in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:327:3
    #21 0x7fa7a932c8fd in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:309:3
    #22 0x7fa7a8ab88f1 in nsThread::ThreadFunc(void*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:442:10
    #23 0x7fa7bcfebabb in _pt_root /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:201:5
    #24 0x7fa7bd67a608 in start_thread /build/glibc-YYA7BZ/glibc-2.31/nptl/pthread_create.c:477:8
    #25 0x7fa7bd243102 in clone /build/glibc-YYA7BZ/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment 1

[Jason Kratzer [:jkratzer]]

Bugmon Analysis:
Unable to reproduce bug using the following builds:
> mozilla-central 20200908215255-dc90a7a18c07
> mozilla-central 20200908030802-80ac8d8c74d5
Removing bugmon keyword as no further action possible.
Please review the bug and re-add the keyword for further analysis.

Comment 2

[Haik Aftandilian [:haik]]

I'm hitting this assertion failure on an Apple Silicon M1 machine with a local build of the beta with the fix for bug 1677854 applied. I've hit the problem a few times browsing twitch.tv.

$ hg parent
changeset:   625607:acd34301a8b6
bookmark:    beta
parent:      625602:5c1464f958ab
user:        Mozilla Releng Treescript <[e-mail]>
date:        Wed Dec 02 16:55:17 2020 +0000
summary:     no bug - Bumping Firefox l10n changesets r=release a=l10n-bump DONTBUILD

Stack:

Assertion failure: aTransform.PreservesAxisAlignedRectangles(), at /Users/haftandilian/r/mu/gfx/layers/composite/LayerManagerComposite.cpp:288
#01: mozilla::layers::LayerManagerComposite::EndTransaction(mozilla::TimeStamp const&, mozilla::layers::LayerManager::EndTransactionFlags)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10afea8]
#02: mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTarg[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10ae9b0]
#03: mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTarg[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10aea4c]
#04: mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTarg[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10aea4c]
#05: mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTarg[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10aea4c]
#06: mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTarg[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10aea4c]
#07: mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::layers::Layer*, mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&, mozilla::gfx::IntRegionTyped<mozilla::LayerPixel>&, mozilla::Maybe<mozilla::gfx::IntRectTyped<mozilla::RenderTarg[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10aea4c]
#08: mozilla::layers::LayerManagerComposite::PostProcessLayers(mozilla::gfx::IntRegionTyped<mozilla::gfx::UnknownUnits>&)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10ae370]
#09: mozilla::layers::LayerManagerComposite::UpdateAndRender()[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10b0148]
#10: mozilla::layers::LayerManagerComposite::EndTransaction(mozilla::TimeStamp const&, mozilla::layers::LayerManager::EndTransactionFlags)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10affe0]
#11: mozilla::layers::CompositorBridgeParent::CompositeToTarget(mozilla::layers::BaseTransactionId<mozilla::VsyncIdType>, mozilla::gfx::DrawTarget*, mozilla::gfx::IntRectTyped<mozilla::gfx::UnknownUnits> const*)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10dbfb8]
#12: mozilla::layers::CompositorVsyncScheduler::Composite(mozilla::VsyncEvent const&)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x10f03cc]
#13: mozilla::detail::RunnableMethodImpl<mozilla::layers::CompositorVsyncScheduler*, void (mozilla::layers::CompositorVsyncScheduler::*)(mozilla::VsyncEvent const&), true, (mozilla::RunnableKind)1, mozilla::VsyncEvent>::Run()[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x110b2d4]
#14: nsThread::ProcessNextEvent(bool, bool*)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x13e7ec]
#15: NS_ProcessNextEvent(nsIThread*, bool)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x143ad4]
#16: mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x7e5b40]
#17: MessageLoop::RunInternal()[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x791414]
#18: MessageLoop::Run()[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x791348]
#19: nsThread::ThreadFunc(void*)[/Users/haftandilian/r/mu/obj-db.noindex/toolkit/library/build/XUL +0x13bdc8]
#20: _pt_root[/Users/haftandilian/r/mu/obj-db.noindex/dist/NightlyDebug.app/Contents/MacOS/libnss3.dylib +0x131848]
#21: _pthread_start[/usr/lib/system/libsystem_pthread.dylib +0x706c]

Comment 3

[Markus Stange [:mstange]]

I would guess that you'd hit this assertion on an Intel machine as well. It doesn't look very arm64-specific.

Comment 4

[Markus Stange [:mstange]]

Also, this is in non-WR code. So if you enable WebRender you should no longer hit the assertion.

Comment 5

[Haik Aftandilian [:haik]]

I marked this as blocking aarch64-macos because I thought it could hinder debugging, but since enabling WebRender is a workaround I'll remove the blocking flag.

Comment 6

[eastern]

Attachment: LayerManagerComposite.patch

I have a patch in which it may fix this crash.

Analysis:
To search AddTransformedRegionRoundIn function in [1], it got two using in LayerManagerComposite.cpp.
We can find that the 3rd parameter in AddTransformedRegionRoundIn is different between two using.
If the Matrix is inverse, it would cause the assert on TransformRect.
If the Matrix is transform2d, it would work finely.

Solution:
Add the checking PreservesAxisAlignedRectangles of Matrix, and pass it by aRoundIn parameter.

Would you help to check it work or not?

[1] https://searchfox.org/mozilla-central/search?q=AddTransformedRegionRoundIn&redirect=false

Comment 7

[Botond Ballo [:botond]]

@eastern, thanks for the proposed patch.

I'm not too familiar with this code, but it looks like the patch changes the behaviour of a function named TransformRectRoundIn() to sometimes not round in, which doesn't sound right (we want functions to do what their name suggests).

I think a better fix may be to add an additional check for inverse.PreservedAxisAlignedRectangles() here.

That said, this code is in the non-WebRender codepath which is slated for removal later this year, so fixes to it are pretty low-priority in the first place.

If you're interested in contributing to Firefox, I encourage you to check out https://codetribute.mozilla.org/, where there are many bugs labelled "good first bug" to choose from.

Comment 8

[eastern]

(In reply to Botond Ballo [:botond] from comment #7)

I think a better fix may be to add an additional check for inverse.PreservedAxisAlignedRectangles() here.

OK, I will look more on it.

That said, this code is in the non-WebRender codepath which is slated for removal later this year, so fixes to it are pretty low-priority in the first place.

what is the plan for removal non-WebRender?

If you're interested in contributing to Firefox, I encourage you to check out https://codetribute.mozilla.org/, where there are many bugs labelled "good first bug" to choose from.
yes, I would check the bugs :)

Comment 9

[Botond Ballo [:botond]]

(In reply to eastern from comment #8)

what is the plan for removal non-WebRender?

There is now a "software WebRender" codepath, which doesn't require GPU support. Machines that don't have the GPU support required for hardware WebRender will fall back to software WebRender.

Comment 10

[Darkspirit]

bug 1727672