1797415 - OOM due to unconstrained memory usage

Status: NEW

(Core :: Graphics: WebRender, defect)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.html

Found while fuzzing m-c 20221020-ca2873779214 (--enable-address-sanitizer --enable-fuzzing)

To reproduce via Grizzly Replay:

$ pip install fuzzfetch grizzly-framework
$ python -m fuzzfetch -a --fuzzing -n firefox
$ ASAN_OPTIONS=hard_rss_limit_mb=6144 python -m grizzly.replay ./firefox/firefox testcase.html

NOTE: Set a reasonable memory limit via ASAN_OPTIONS=hard_rss_limit_mb=# to avoid system OOMs.

This test case does not trigger an OOM on Chrome.

HEAP PROFILE at RSS 12385Mb
Live Heap Allocations: 11587412350 bytes in 317470 chunks; quarantined: 260355704 bytes in 168759 chunks; 497360 other chunks; total chunks: 983589; showing top 90% (at most 20 unique contexts)
11432886272 byte(s) (98%) in 689 allocation(s)
    #0 0x56339986f206 in __interceptor_realloc /builds/worker/fetches/llvm-project/compiler-rt/lib/asan/asan_malloc_linux.cpp:85:3
    #1 0x7f5f9f090ac2 in Texture::allocate(bool, int, int) /builds/worker/checkouts/gecko/gfx/wr/swgl/src/gl.cc:509:32
    #2 0x7f5f9f07a5df in set_tex_storage(Texture&, unsigned int, int, int, void*, int, int, int) /builds/worker/checkouts/gecko/gfx/wr/swgl/src/gl.cc:1716:10
    #3 0x7f5f9f07a0b6 in TexStorage2D /builds/worker/checkouts/gecko/gfx/wr/swgl/src/gl.cc:1732:3
    #4 0x7f5f9f07b0d1 in TexImage2D /builds/worker/checkouts/gecko/gfx/wr/swgl/src/gl.cc:1822:3
    #5 0x7f5f9f05c2fa in _$LT$swgl..swgl_fns..Context$u20$as$u20$gleam..gl..Gl$GT$::tex_image_2d::h096404e29065fde1 /builds/worker/checkouts/gecko/gfx/wr/swgl/src/swgl_fns.rs:997:13
    #6 0x7f5f9e49fb80 in webrender::device::gl::Device::create_texture::h802b74acde894c22 /builds/worker/checkouts/gecko/gfx/wr/webrender/src/device/gl.rs:2584:13
    #7 0x7f5f9e853b41 in webrender::renderer::Renderer::update_texture_cache::_$u7b$$u7b$closure$u7d$$u7d$::h0853617505dc9afb /builds/worker/checkouts/gecko/gfx/wr/webrender/src/renderer/mod.rs:1845:29
    #8 0x7f5f9e853b41 in core::option::Option$LT$T$GT$::unwrap_or_else::h7da79574801223a9 /builds/worker/fetches/rust/library/core/src/option.rs:825:21
    #9 0x7f5f9e853b41 in webrender::renderer::Renderer::update_texture_cache::h2ae24f8985f243b9 /builds/worker/checkouts/gecko/gfx/wr/webrender/src/renderer/mod.rs:1844:43
    #10 0x7f5f9e82fef7 in webrender::renderer::Renderer::render_impl::h18ab4a8c1e63f677 /builds/worker/checkouts/gecko/gfx/wr/webrender/src/renderer/mod.rs:1437:13
    #11 0x7f5f9e82b94d in webrender::renderer::Renderer::render::h02d2d54934f8ddbb /builds/worker/checkouts/gecko/gfx/wr/webrender/src/renderer/mod.rs:1199:30
    #12 0x7f5f9dc4db79 in wr_renderer_render /builds/worker/checkouts/gecko/gfx/webrender_bindings/src/bindings.rs:619:11
    #13 0x7f5f8ebea191 in mozilla::wr::RendererOGL::UpdateAndRender(mozilla::Maybe<mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> > const&, mozilla::Maybe<mozilla::wr::ImageFormat> const&, mozilla::Maybe<mozilla::Range<unsigned char> > const&, bool*, mozilla::wr::RendererStats*) /builds/worker/checkouts/gecko/gfx/webrender_bindings/RendererOGL.cpp:186:19
    #14 0x7f5f8ebe84d8 in mozilla::wr::RenderThread::UpdateAndRender(mozilla::wr::WrWindowId, mozilla::layers::BaseTransactionId<mozilla::VsyncIdType> const&, mozilla::TimeStamp const&, bool, mozilla::Maybe<mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> > const&, mozilla::Maybe<mozilla::wr::ImageFormat> const&, mozilla::Maybe<mozilla::Range<unsigned char> > const&, bool*) /builds/worker/checkouts/gecko/gfx/webrender_bindings/RenderThread.cpp:565:31
    #15 0x7f5f8ebe756f in mozilla::wr::RenderThread::HandleFrameOneDoc(mozilla::wr::WrWindowId, bool) /builds/worker/checkouts/gecko/gfx/webrender_bindings/RenderThread.cpp:411:3
    #16 0x7f5f8ec04826 in applyImpl<mozilla::wr::RenderThread, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool), StoreCopyPassByConstLRef<mozilla::wr::WrWindowId>, StoreCopyPassByConstLRef<bool>, 0UL, 1UL> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1147:12
    #17 0x7f5f8ec04826 in apply<mozilla::wr::RenderThread, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool)> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1153:12
    #18 0x7f5f8ec04826 in mozilla::detail::RunnableMethodImpl<mozilla::wr::RenderThread*, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool), true, (mozilla::RunnableKind)0, mozilla::wr::WrWindowId, bool>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1200:13
    #19 0x7f5f8bfc1d7e in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:1199:16
    #20 0x7f5f8bfcb9d4 in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/checkouts/gecko/xpcom/threads/nsThreadUtils.cpp:465:10
    #21 0x7f5f8d7757e1 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/checkouts/gecko/ipc/glue/MessagePump.cpp:330:5
    #22 0x7f5f8d5f1351 in RunInternal /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:381:10
    #23 0x7f5f8d5f1351 in RunHandler /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:374:3
    #24 0x7f5f8d5f1351 in MessageLoop::Run() /builds/worker/checkouts/gecko/ipc/chromium/src/base/message_loop.cc:356:3
    #25 0x7f5f8bfb8ec8 in nsThread::ThreadFunc(void*) /builds/worker/checkouts/gecko/xpcom/threads/nsThread.cpp:384:10
    #26 0x7f5fb3eaeb7e in _pt_root /builds/worker/checkouts/gecko/nsprpub/pr/src/pthreads/ptthread.c:201:5
    #27 0x7f5fb48ea608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8

Comment 1

[Jamie Nicol [:jnicol]]

Glenn, could the backdrop filter or the mask in the testcase be causing us to allocate too many texture cache entries?

Comment 2

[Glenn Watson [:gw]]

It's possible, yes. I'll investigate it this week.