1809309 - Hit MOZ_CRASH(no entry found for key) at gfx/wr/webrender/src/scene_building.rs:141

Status: REOPENED

(Core :: Graphics: WebRender, defect)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.html

Found while fuzzing m-c 20221127-f49e8eca9e34 (--enable-address-sanitizer --enable-fuzzing)

To reproduce via Grizzly Replay:

$ pip install fuzzfetch grizzly-framework
$ python -m fuzzfetch -a --fuzzing -n firefox
$ python -m grizzly.replay ./firefox/firefox testcase.html --xvfb

Hit MOZ_CRASH(no entry found for key) at gfx/wr/webrender/src/scene_building.rs:141

#0 0x7fdcbefca3f9 in MOZ_Crash /builds/worker/workspace/obj-build/dist/include/mozilla/Assertions.h:261:3
#1 0x7fdcbefca3f9 in RustMozCrash /builds/worker/checkouts/gecko/mozglue/static/rust/wrappers.cpp:18:3
#2 0x7fdcbefca2c0 in mozglue_static::panic_hook::h2528d155d73bb4bb /builds/worker/checkouts/gecko/mozglue/static/rust/lib.rs:91:9
#3 0x7fdcbefc8fe5 in core::ops::function::Fn::call::hc91d87086350bb43 /builds/worker/fetches/rust/library/core/src/ops/function.rs:78:5
#4 0x7fdcc2d6900b in std::panicking::rust_panic_with_hook::hb95930056730415d (/home/user/workspace/browsers/m-c-20230109162059-fuzzing-asan-opt/libxul.so+0x27e5e00b) (BuildId: 6d9a393173a32f5d05e82cf8cdbe47b055bbd218)
#5 0x7fdcc2d8d496 in std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h251d4677403105eb std.82f3c14a-cgu.8
#6 0x7fdcc2d8d28b in std::sys_common::backtrace::__rust_end_short_backtrace::h4aa72274704f4358 std.82f3c14a-cgu.8
#7 0x7fdcc2d68b81 in rust_begin_unwind (/home/user/workspace/browsers/m-c-20230109162059-fuzzing-asan-opt/libxul.so+0x27e5db81) (BuildId: 6d9a393173a32f5d05e82cf8cdbe47b055bbd218)
#8 0x7fdcc2dd96b2 in core::panicking::panic_fmt::h8c57bd6922066c10 (/home/user/workspace/browsers/m-c-20230109162059-fuzzing-asan-opt/libxul.so+0x27ece6b2) (BuildId: 6d9a393173a32f5d05e82cf8cdbe47b055bbd218)
#9 0x7fdcc2dd08f0 in core::panicking::panic_display::hb307d970a692863b core.509e1a9a-cgu.15
#10 0x7fdcc2dd089a in core::panicking::panic_str::h844e3d6281227297 core.509e1a9a-cgu.15
#11 0x7fdcc2dd06d5 in core::option::expect_failed::h630f7be4efe18631 (/home/user/workspace/browsers/m-c-20230109162059-fuzzing-asan-opt/libxul.so+0x27ec56d5) (BuildId: 6d9a393173a32f5d05e82cf8cdbe47b055bbd218)
#12 0x7fdcbdf8d3a7 in core::option::Option$LT$T$GT$::expect::hd2b582fa73e75548 /builds/worker/fetches/rust/library/core/src/option.rs:741:21
#13 0x7fdcbdf8d3a7 in _$LT$std..collections..hash..map..HashMap$LT$K$C$V$C$S$GT$$u20$as$u20$core..ops..index..Index$LT$$RF$Q$GT$$GT$::index::h54b726a8489b1ba6 /builds/worker/fetches/rust/library/std/src/collections/hash/map.rs:1340:9
#14 0x7fdcbdf8d3a7 in webrender::scene_building::NodeIdToIndexMapper::get_spatial_node_index::h611f76b7aa0cc002 /builds/worker/checkouts/gecko/gfx/wr/webrender/src/scene_building.rs:141:9
#15 0x7fdcbdf8d3a7 in webrender::scene_building::SceneBuilder::get_space::hc9d9d8678d8fd1ee /builds/worker/checkouts/gecko/gfx/wr/webrender/src/scene_building.rs:1166:9
#16 0x7fdcbdf92c90 in webrender::scene_building::SceneBuilder::build_item::hd3c107de9a4dcf7e /builds/worker/checkouts/gecko/gfx/wr/webrender/src/scene_building.rs:1365:42
#17 0x7fdcbdf5fcb7 in webrender::scene_building::SceneBuilder::build_all::h3ea8e50cd8c6a2b8 /builds/worker/checkouts/gecko/gfx/wr/webrender/src/scene_building.rs:885:25
#18 0x7fdcbdf5fcb7 in webrender::scene_building::SceneBuilder::build::h73a9a5ec3c8c3b20 /builds/worker/checkouts/gecko/gfx/wr/webrender/src/scene_building.rs:592:9
#19 0x7fdcbdf41aa3 in webrender::scene_builder_thread::SceneBuilderThread::process_transaction::hb36fb89abaf29f71 /builds/worker/checkouts/gecko/gfx/wr/webrender/src/scene_builder_thread.rs:597:25
#20 0x7fdcbdeff4cd in webrender::scene_builder_thread::SceneBuilderThread::run::_$u7b$$u7b$closure$u7d$$u7d$::hc9821c0cff77b3ff /builds/worker/checkouts/gecko/gfx/wr/webrender/src/scene_builder_thread.rs:314:36
#21 0x7fdcbdeff4cd in core::iter::adapters::map::map_try_fold::_$u7b$$u7b$closure$u7d$$u7d$::he827f541c9b9c79f /builds/worker/fetches/rust/library/core/src/iter/adapters/map.rs:91:28
#22 0x7fdcbdeff4cd in core::iter::traits::iterator::Iterator::try_fold::hc19df34b8ad388ec /builds/worker/fetches/rust/library/core/src/iter/traits/iterator.rs:2238:21
#23 0x7fdcbdeff4cd in _$LT$core..iter..adapters..map..Map$LT$I$C$F$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::try_fold::h339730f05871c97e /builds/worker/fetches/rust/library/core/src/iter/adapters/map.rs:117:9
#24 0x7fdcbdeff4cd in _$LT$I$u20$as$u20$alloc..vec..in_place_collect..SpecInPlaceCollect$LT$T$C$I$GT$$GT$::collect_in_place::h94f9d9bfa25110cb /builds/worker/fetches/rust/library/alloc/src/vec/in_place_collect.rs:257:13
#25 0x7fdcbdeff4cd in alloc::vec::in_place_collect::_$LT$impl$u20$alloc..vec..spec_from_iter..SpecFromIter$LT$T$C$I$GT$$u20$for$u20$alloc..vec..Vec$LT$T$GT$$GT$::from_iter::h90a46b5acb13d0b0 /builds/worker/fetches/rust/library/alloc/src/vec/in_place_collect.rs:181:19
#26 0x7fdcbdeff4cd in _$LT$alloc..vec..Vec$LT$T$GT$$u20$as$u20$core..iter..traits..collect..FromIterator$LT$T$GT$$GT$::from_iter::h652eb07ae74d692b /builds/worker/fetches/rust/library/alloc/src/vec/mod.rs:2757:9
#27 0x7fdcbdeff4cd in core::iter::traits::iterator::Iterator::collect::h13b1ef7c1ecc0191 /builds/worker/fetches/rust/library/core/src/iter/traits/iterator.rs:1836:9
#28 0x7fdcbdeff4cd in webrender::scene_builder_thread::SceneBuilderThread::run::hcb320a3db8a199ba /builds/worker/checkouts/gecko/gfx/wr/webrender/src/scene_builder_thread.rs:313:67
#29 0x7fdcbd3133bc in webrender::renderer::init::create_webrender_instance::_$u7b$$u7b$closure$u7d$$u7d$::h97f727203a56cbc3 /builds/worker/checkouts/gecko/gfx/wr/webrender/src/renderer/init.rs:590:9
#30 0x7fdcbd3133bc in std::sys_common::backtrace::__rust_begin_short_backtrace::hf91df755ed112be5 /builds/worker/fetches/rust/library/std/src/sys_common/backtrace.rs:121:18
#31 0x7fdcbd36fb64 in std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h97d7d6096729ab2d /builds/worker/fetches/rust/library/std/src/thread/mod.rs:551:17
#32 0x7fdcbd36fb64 in _$LT$core..panic..unwind_safe..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h39b2df182df0a8cf /builds/worker/fetches/rust/library/core/src/panic/unwind_safe.rs:271:9
#33 0x7fdcbd36fb64 in std::panicking::try::do_call::h7c23f48793e57d45 /builds/worker/fetches/rust/library/std/src/panicking.rs:483:40
#34 0x7fdcbd36fb64 in std::panicking::try::h247f9e05fab421a9 /builds/worker/fetches/rust/library/std/src/panicking.rs:447:19
#35 0x7fdcbd36fb64 in std::panic::catch_unwind::h5d30b680f2788c52 /builds/worker/fetches/rust/library/std/src/panic.rs:137:14
#36 0x7fdcbd36fb64 in std::thread::Builder::spawn_unchecked_::_$u7b$$u7b$closure$u7d$$u7d$::hfc0a7f8814e99f2a /builds/worker/fetches/rust/library/std/src/thread/mod.rs:550:30
#37 0x7fdcbd36fb64 in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hfa821e453454aaa7 /builds/worker/fetches/rust/library/core/src/ops/function.rs:251:5
#38 0x7fdcc2d964f2 in std::sys::unix::thread::Thread::new::thread_start::h053bd8e54c50a3de std.82f3c14a-cgu.9
#39 0x7fdcce0b1b42 in start_thread nptl/pthread_create.c:442:8
#40 0x7fdcce1439ff  misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Comment 1

[Tyson Smith [:tsmith]]

A Pernosco session is available here: https://pernos.co/debug/xfmQc8sNQYERihhUak8JXw/index.html

Comment 2

[Bugmon [:jkratzer for issues]]

Verified bug as reproducible on mozilla-central 20230109212101-329b80a0d033.
The bug appears to have been introduced in the following build range:

Start: 5936168c80d1f6b8a55f7f528b0851e75e90660d (20220906092501)
End: d1b399bcd0474869d29804c13b2145a6a8b645da (20220906120315)
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=5936168c80d1f6b8a55f7f528b0851e75e90660d&tochange=d1b399bcd0474869d29804c13b2145a6a8b645da

Comment 3

[Frédéric Wang (:fredw)]

Attachment: testcase (s/msub/mrow/)

Comment 4

[Frédéric Wang (:fredw)]

mmh, it seems my comment was lost:

"I think this is an instance of the cases where the legacy MathML "invalid markup" message was hiding bugs found by fuzzers (this message was removed in https://hg.mozilla.org/integration/autoland/rev/69aab0d556424c65172360f3c2f02c2809e6522c but can be reactivated via the mathml.error_message_layout_for_invalid_markup.disabled pref)

Anyway, here is an alternative testcase with s/msub/mrow/ so that the missing child of the msub element does not cause an invalid markup message. It crashes for me at 1ff7828b2117371e1d2536dfb5ff9d7ca7e057be It would be a good idea to rebisect with that testcase instead..."

Comment 5

[Tyson Smith [:tsmith]]

Alright, let's see what bugmon says now.

Comment 6

[Bugmon [:jkratzer for issues]]

Verified bug as reproducible on mozilla-central 20230110214526-9231302514fc.
Unable to bisect testcase (Testcase reproduces on start build!):

Start: bb37e7d6382b8647a9567947d18dce1e61e670e6 (20220112035347)
End: f49e8eca9e344e5d8b9a5e67ff5859ba3afc3a4d (20221127212619)
BuildFlags: BuildFlags(asan=True, tsan=False, debug=False, fuzzing=True, coverage=False, valgrind=False, no_opt=False, fuzzilli=False, nyx=False)

Comment 7

[Ashley Hale [:ahale]]

Triage - I am rating this S3 while discussion is ongoing in the bug, please NI me for a re-rating of the bug if concerns arise - as it stands this looks like an undesired but safe crash?

Comment 8

[Bugmon [:jkratzer for issues]]

Bugmon was unable reproduce this issue.
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.

Comment 9

[Jason Kratzer [:jkratzer]]

A change to the Taskcluster build definitions over the weekend caused Bugmon to fail when reproducing issues. This issue has been corrected. Re-enabling bugmon.

Comment 10

[Bugmon [:jkratzer for issues]]

Testcase crashes using the initial build (mozilla-central 20230128211106-f4f63f0138fe) but not with tip (mozilla-central 20240127092204-0452ed2e98ac.)

The bug appears to have been fixed in the following build range:

Start: 2437c2ca5bec35fe4ab02eb938e0e02457cd079b (20240123142542)
End: f3efca74da0f43269bd8ac07e2a5d27e89c4d7c3 (20240123145016)
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=2437c2ca5bec35fe4ab02eb938e0e02457cd079b&tochange=f3efca74da0f43269bd8ac07e2a5d27e89c4d7c3

tsmith, can you confirm that the above bisection range is responsible for fixing this issue?
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Is it worth landing the testcase from this bug or do the other ones landed in bug 1874826 cover it sufficiently?

Comment 12

[Frédéric Wang (:fredw)]

I'll check, but actually I'm not sure if 1874826 is responsible for the fix. Emilio's patches may be candidates too.

Comment 13

[Frédéric Wang (:fredw)]

(In reply to Bugmon [:jkratzer for issues] from comment #10)

Testcase crashes using the initial build (mozilla-central 20230128211106-f4f63f0138fe) but not with tip (mozilla-central 20240127092204-0452ed2e98ac.)

The bug appears to have been fixed in the following build range:

Start: 2437c2ca5bec35fe4ab02eb938e0e02457cd079b (20240123142542)
End: f3efca74da0f43269bd8ac07e2a5d27e89c4d7c3 (20240123145016)
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=2437c2ca5bec35fe4ab02eb938e0e02457cd079b&tochange=f3efca74da0f43269bd8ac07e2a5d27e89c4d7c3

tsmith, can you confirm that the above bisection range is responsible for fixing this issue?
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.

For this range, I'm hitting a different assertion failure MOZ_ASSERT(IsAncestor(aOne, aTwo) || IsAncestor(aTwo, aOne)) from https://searchfox.org/mozilla-central/rev/6b8a3f804789fb865f42af54e9d2fef9dd3ec74d/layout/painting/nsDisplayList.h#203

At least I noticed the following:

crashes:
https://hg.mozilla.org/integration/autoland/rev/2437c2ca5bec35fe4ab02eb938e0e02457cd079b

non-determistic crashes:
https://hg.mozilla.org/integration/autoland/rev/5c38b3e5008b25db6a3c20acc5df4e5e10c7af52
https://hg.mozilla.org/integration/autoland/rev/f3efca74da0f43269bd8ac07e2a5d27e89c4d7c3

So maybe bugmon may just have been confused by 1) a different assert 2) the crash becoming non-determinstic

Comment 14

[Frédéric Wang (:fredw)]

Attachment: moz-assert-is-ancestor.html

This is a reduced testcase for the IsAncestor(aOne, aTwo) which happens in the latest nightly. Not sure I should open a separate bug for that, it seems we already have similar reports at bug 1427792, bug 1729589, bug 1765168, bug 1787690, bug 1826032 and bug 1855249.

Comment 15

[Frédéric Wang (:fredw)]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #11)

Is it worth landing the testcase from this bug or do the other ones landed in bug 1874826 cover it sufficiently?

As explained above, I can't reproduce the original MOZ_CRASH(no entry found for key) in the range mentioned in comment 10, instead I see a different assert due to a subset of the original testcase (comment 14).

In any case it's not obvious to me what the connection to bug 1874826 is, unless maybe opening/closing dialog changes containment.

I guess it would help to get a bisection of when this MOZ_CRASH(no entry found for key) actually disappeared.

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Jason, thoughts?

Comment 17

[Tyson Smith [:tsmith]]

Re comment 14: tnikkel would be the person to ask.

Comment 18

[Jason Kratzer [:jkratzer]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #16)

Jason, thoughts?

So unfortunately, I have been unable to reproduce this locally using either testcase and the build in comment 0. In regards to the bisection range, bugmon does not try to differentiate between crashes (since the signature may change over time).

:fredw, if you can give me any suggestions on what I can do to trigger this locally I can give you a more accurate bisection range.

Comment 19

[Timothy Nikkel (:tnikkel)]

(In reply to Frédéric Wang (:fredw) from comment #14)

Created attachment 9378690 [details]
moz-assert-is-ancestor.html

This is a reduced testcase for the IsAncestor(aOne, aTwo) which happens in the latest nightly. Not sure I should open a separate bug for that, it seems we already have similar reports at bug 1427792, bug 1729589, bug 1765168, bug 1787690, bug 1826032 and bug 1855249.

Looks like the testcase uses clip-path and dialog which is a pretty common pattern for the testcases in those bugs that trigger this assert. I don't think this testcase adds anything to the existing testcases in those bugs so I don't think we need to open a new bug. Thanks for checking.

Comment 20

[Frédéric Wang (:fredw)]

(In reply to Jason Kratzer [:jkratzer] from comment #18)

(In reply to Ryan VanderMeulen [:RyanVM] from comment #16)

Jason, thoughts?

So unfortunately, I have been unable to reproduce this locally using either testcase and the build in comment 0. In regards to the bisection range, bugmon does not try to differentiate between crashes (since the signature may change over time).

:fredw, if you can give me any suggestions on what I can do to trigger this locally I can give you a more accurate bisection range.

Right, it's no longer happening for me with attachment 9311512 [details] at https://hg.mozilla.org/mozilla-central/rev/fce00cd55ffb although I still get an intermittent MOZ_ASSERT(IsAncestor(aOne, aTwo) || IsAncestor(aTwo, aOne)).

However, I believe the MOZ_ASSERT(no entry found for key) was reproducible with attachment 9311512 [details] at the the git hash I provided one year ago in comment 4 i.e. https://hg.mozilla.org/mozilla-central/rev/370b88940241e124a82dee16c6fce90ee25af36b (note that those from comment 6 happened earlier). Isn't it the case for you?

From these two commits, it should be possible to bisect when this was actually fixed, based on the actual assert message, not just the crash.

Comment 21

[Jason Kratzer [:jkratzer]]

I'm not sure what changed but using attachment 9311512 [details] I can once again reproduce the original issue listed in comment 0.

➜  test python -m grizzly.replay --xvfb -p prefs.js /home/jkratzer/.cache/autobisect/builds/firefox-m-c-linux-fuzzing-asan-opt-552269a748b3/firefox testcase.html 
/home/jkratzer/.local/lib/python3.10/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (5.2.0)/charset_normalizer (2.0.12) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported "
[2024-02-21 10:31:14] Starting Grizzly Replay
[2024-02-21 10:31:14] Running browser headless (xvfb)
[2024-02-21 10:31:14] Ignoring: log-limit, timeout
[2024-02-21 10:31:14] Using time limit: 30s, timeout: 45s
[2024-02-21 10:31:14] Repeat: 1, Minimum crashes: 1, Relaunch 1
[2024-02-21 10:31:18] Running test (1/1)...
[2024-02-21 10:31:19] Processing result...
[2024-02-21 10:31:19] Result: Hit MOZ_CRASH(no entry found for key) at gfx/wr/webrender/src/scene_building.rs:141 (0f8950d2:4f8c0368)
[2024-02-21 10:31:19] Results successfully reproduced
[2024-02-21 10:31:19] Shutting down...
[2024-02-21 10:31:19] Done.
➜  test cat /home/jkratzer/.cache/autobisect/builds/firefox-m-c-linux-fuzzing-asan-opt-552269a748b3/firefox.fuzzmanagerconf 
[Main]
platform = x86-64
product = mozilla-central
product_version = 20240221-552269a748b3
os = linux

[Metadata]
pathprefix = /builds/worker/checkouts/gecko
buildtype = fuzzing-asan-opt

Comment 22

[Frédéric Wang (:fredw)]

Unassigning, as the original report that this was due to mathml is likely wrong.