1700248 - Hit MOZ_CRASH(internal error: entered unreachable code) at gfx/wr/webrender/src/render_task.rs:586

Status: RESOLVED DUPLICATE of bug 1695016

(Core :: Graphics: WebRender, defect, P3)

Description

[Tyson Smith [:tsmith]]

First found while fuzzing m-c 20210228-f875a4ffd653. We do not have a reliable test case that will easily reduce.

Hit MOZ_CRASH(internal error: entered unreachable code) at gfx/wr/webrender/src/render_task.rs:586

#0 0x7f46a6099cf5 in MOZ_Crash /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:254:3
#1 0x7f46a6099cf5 in RustMozCrash src/mozglue/static/rust/wrappers.cpp:17:3
#2 0x7f46a6099ca4 in mozglue_static::panic_hook::hbbc7bc8518f2f09c src/mozglue/static/rust/lib.rs:89:9
#3 0x7f46a609967b in core::ops::function::Fn::call::h74ce577ceebae4e6 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/ops/function.rs:70:5
#4 0x7f46a70a6195 in std::panicking::rust_panic_with_hook::hb27ea14285131c61 /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/panicking.rs:595:17
#5 0x7f46a70a5c86 in std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::hc552fcee62aad17f /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/panicking.rs:495:13
#6 0x7f46a70a20db in std::sys_common::backtrace::__rust_end_short_backtrace::hb9f0aa9a78e885a0 /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/sys_common/backtrace.rs:141:18
#7 0x7f46a70a5c18 in rust_begin_unwind /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/panicking.rs:493:5
#8 0x7f46a710f3e0 in core::panicking::panic_fmt::h12ac4570ea43d06f /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/core/src/panicking.rs:92:14
#9 0x7f46a710f32c in core::panicking::panic::h72bd72f6f4a70105 /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/core/src/panicking.rs:50:5
#10 0x7f46a589f6d9 in webrender::render_task::RenderTaskKind::new_mask::h8d1b2c0a3e464a83 src/gfx/wr/webrender/src/render_task.rs:586:33
#11 0x7f46a585fae1 in webrender::prepare::update_clip_task::h28ddebc2d1f7553a src/gfx/wr/webrender/src/prepare.rs:1514:28
#12 0x7f46a584f786 in webrender::prepare::prepare_prim_for_render::h362c35786dfa6192 src/gfx/wr/webrender/src/prepare.rs:253:13
#13 0x7f46a584f786 in webrender::prepare::prepare_primitives::h0f0e776ba10f0f7d src/gfx/wr/webrender/src/prepare.rs:119:16
#14 0x7f46a57f2448 in webrender::frame_builder::FrameBuilder::build_layer_screen_rects_and_cull_layers::h783c076f61b007f9 src/gfx/wr/webrender/src/frame_builder.rs:478:17
#15 0x7f46a57f2448 in webrender::frame_builder::FrameBuilder::build::h0bdbf9dfa48a90e6 src/gfx/wr/webrender/src/frame_builder.rs:570:9
#16 0x7f46a58768be in webrender::render_backend::Document::build_frame::he2343d93fff0c4ed src/gfx/wr/webrender/src/render_backend.rs:622:25
#17 0x7f46a58877db in webrender::render_backend::RenderBackend::update_document::he42f3d86d2ddc615 src/gfx/wr/webrender/src/render_backend.rs:1508:41
#18 0x7f46a587dbc6 in webrender::render_backend::RenderBackend::prepare_transactions::hc85c91340bbe6ac7 src/gfx/wr/webrender/src/render_backend.rs:1362:28
#19 0x7f46a587dbc6 in webrender::render_backend::RenderBackend::process_api_msg::h68b0322d0239fcbc src/gfx/wr/webrender/src/render_backend.rs:1218:17
#20 0x7f46a5663a2d in webrender::render_backend::RenderBackend::run::h031c3d81b51f8ddd src/gfx/wr/webrender/src/render_backend.rs:894:21
#21 0x7f46a5663a2d in webrender::renderer::Renderer::new::_$u7b$$u7b$closure$u7d$$u7d$::h78785921f24c3e5b src/gfx/wr/webrender/src/renderer/mod.rs:1278:13
#22 0x7f46a5663a2d in std::sys_common::backtrace::__rust_begin_short_backtrace::ha1c6115d5f09754b /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125:18
#23 0x7f46a5684879 in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h8a21d5b4f8815dfa /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474:17
#24 0x7f46a5684879 in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::haf44b2a8608c2080 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panic.rs:322:9
#25 0x7f46a5684879 in std::panicking::try::do_call::hb87d5846e8b53b30 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panicking.rs:379:40
#26 0x7f46a5684879 in std::panicking::try::h0338be5c16956d70 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panicking.rs:343:19
#27 0x7f46a5684879 in std::panic::catch_unwind::h35071a840a72e312 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panic.rs:396:14
#28 0x7f46a5684879 in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h8d028a379752c475 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473:30
#29 0x7f46a5684879 in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h7654f74ec7a6776e /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
#30 0x7f46a70b65a9 in _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h9ed215ba67984d70 /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/alloc/src/boxed.rs:1328:9
#31 0x7f46a70b65a9 in _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hcece06e1fe04906f /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/alloc/src/boxed.rs:1328:9
#32 0x7f46a70b65a9 in std::sys::unix::thread::Thread::new::thread_start::h6e82a4b7be15319a /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/sys/unix/thread.rs:71:17
#33 0x7f46b50f3608 in start_thread /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:477:8
#34 0x7f46b4cbc292 in clone /build/glibc-eX1tMB/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment 1

[Tyson Smith [:tsmith]]

A Pernosco session is available here: https://pernos.co/debug/SNCI59FWsAbXx9jttP1tKQ/index.html

Comment 2

[Andrew Osmond [:aosmond] (he/him)]

https://searchfox.org/mozilla-central/rev/368607c4cd5be547021945e4ae60e8eb4365b3c4/gfx/wr/webrender/src/render_task.rs#583

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

clip_task.kind is Border(..)

Comment 4

[Glenn Watson [:gw]]

It looks like this is occurring because the fuzzing test case creates > 2^16 render tasks. The index was changed from a u32 to a u16 in https://phabricator.services.mozilla.com/D105986.

If there are > 65536 render tasks, we're not going to be able to draw it in any reasonable time (if at all), either due to the number of render target allocations / swaps, or GPU memory exhaustion, or simply due to CPU time to process and batch all that work. To give an idea of scale, most pages contain somewhere in the range of a few dozen render tasks. A page with hundreds or low thousands of tasks would be an extreme stress case.

So, options I can see are (from least work to most work):

• Make it a u32 index again, and pay the performance impact.
• panic the GPU process if a page encounters > 2^16 render tasks (or maybe even something more like 2^13 render tasks).
• Handle this gracefully by making render task allocation fallible (there are lots of call sites that would need updating).

Thoughts?

Comment 5

[Jeff Muizelaar [:jrmuizel]]

I'm not sure what the test case looks like. My suggestion is that we try to prevent this early if practical and otherwise panic in the GPU process essentially treating it like an OOM

Comment 6

[Nicolas Silva [:nical]]

See 1695016 comment 8 which is essentially the same thing.

I don't think it is possible to properly handle this at frame building time without risking content corrupting the chrome. Ideally things like this would be caught during displaylist building so that we can crash the content process instead of the parent/gpu process.