1700494 - Assertion failure: [GFX1 39]: Failed to get ScaledFont entry for FontInstanceKey 24, at src/gfx/2d/Logging.h:754

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.html

Found while fuzzing m-c 20210121-e5e734f4fc19 (--enable-debug --enable-fuzzing)

Assertion failure: [GFX1 39]: Failed to get ScaledFont entry for FontInstanceKey 24, at src/gfx/2d/Logging.h:754

#0 0x7fe1b2a77a3f in mozilla::gfx::Log<1, mozilla::gfx::CriticalLogger>::WriteLog(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/gfx/2d/Logging.h:755:9
#1 0x7fe1b2a7794e in mozilla::gfx::Log<1, mozilla::gfx::CriticalLogger>::Flush() src/gfx/2d/Logging.h:277:7
#2 0x7fe1b302032a in ~Log /builds/worker/workspace/obj-build/dist/include/mozilla/gfx/Logging.h:270:12
#3 0x7fe1b302032a in GetScaledFont src/gfx/webrender_bindings/Moz2DImageRenderer.cpp:286:5
#4 0x7fe1b302032a in Moz2DRenderCallback src/gfx/webrender_bindings/Moz2DImageRenderer.cpp:421:11
#5 0x7fe1b302032a in wr_moz2d_render_cb src/gfx/webrender_bindings/Moz2DImageRenderer.cpp:471:10
#6 0x7fe1b937dde0 in webrender_bindings::moz2d_renderer::rasterize_blob::_$u7b$$u7b$closure$u7d$$u7d$::ha4f9af9216b8a2e2 src/gfx/webrender_bindings/src/moz2d_renderer.rs:644:16
#7 0x7fe1b937dde0 in webrender_bindings::moz2d_renderer::autoreleasepool::h9f786da46ea8e00b src/gfx/webrender_bindings/src/moz2d_renderer.rs:625:9
#8 0x7fe1b937dde0 in webrender_bindings::moz2d_renderer::rasterize_blob::hee70ef9fdb4716f4 src/gfx/webrender_bindings/src/moz2d_renderer.rs:642:18
#9 0x7fe1b936092b in core::ops::function::Fn::call::hdfd7826127344c46 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/ops/function.rs:70:5
#10 0x7fe1b936092b in core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnMut$LT$A$GT$$u20$for$u20$$RF$F$GT$::call_mut::h9ef54f4ef0c7d75f /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/ops/function.rs:247:13
#11 0x7fe1b936092b in core::ops::function::impls::_$LT$impl$u20$core..ops..function..FnOnce$LT$A$GT$$u20$for$u20$$RF$mut$u20$F$GT$::call_once::h09cbcd7f091e7944 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/ops/function.rs:280:13
#12 0x7fe1b936092b in core::option::Option$LT$T$GT$::map::hf6fe12d1954be63b /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/option.rs:453:29
#13 0x7fe1b936092b in _$LT$core..iter..adapters..map..Map$LT$I$C$F$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::next::h10e03a6fc06006d2 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/iter/adapters/map.rs:99:9
#14 0x7fe1b936092b in rayon::iter::plumbing::Folder::consume_iter::h345a4385457c24ce src/third_party/rust/rayon/src/iter/plumbing/mod.rs:178:21
#15 0x7fe1b936092b in _$LT$rayon..iter..map..MapFolder$LT$C$C$F$GT$$u20$as$u20$rayon..iter..plumbing..Folder$LT$T$GT$$GT$::consume_iter::h24532e59471931c5 src/third_party/rust/rayon/src/iter/map.rs:248:21
#16 0x7fe1b936092b in rayon::iter::plumbing::Producer::fold_with::hd8ef3272d5e384bb src/third_party/rust/rayon/src/iter/plumbing/mod.rs:110:9
#17 0x7fe1b936092b in rayon::iter::plumbing::bridge_producer_consumer::helper::hc4d3c979010b9aa4 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:438:13
#18 0x7fe1b93117c9 in rayon::iter::plumbing::bridge_producer_consumer::helper::_$u7b$$u7b$closure$u7d$$u7d$::hea893d99d2794791 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:427:21
#19 0x7fe1b93117c9 in rayon_core::join::join_context::call_b::_$u7b$$u7b$closure$u7d$$u7d$::h49bf5a792680eaa9 src/third_party/rust/rayon-core/src/join/mod.rs:129:25
#20 0x7fe1b93117c9 in rayon_core::job::StackJob$LT$L$C$F$C$R$GT$::run_inline::h1e263ba69f795e5f src/third_party/rust/rayon-core/src/job.rs:97:9
#21 0x7fe1b93117c9 in rayon_core::join::join_context::_$u7b$$u7b$closure$u7d$$u7d$::h20cb11dcd0d2a430 src/third_party/rust/rayon-core/src/join/mod.rs:158:36
#22 0x7fe1b9360b56 in rayon_core::registry::in_worker::h7ef39ba38031c680 src/third_party/rust/rayon-core/src/registry.rs:879:13
#23 0x7fe1b9360b56 in rayon_core::join::join_context::h84f142bc3fc0819c src/third_party/rust/rayon-core/src/join/mod.rs:132:5
#24 0x7fe1b9360b56 in rayon::iter::plumbing::bridge_producer_consumer::helper::hc4d3c979010b9aa4 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:416:47
#25 0x7fe1b93117c9 in rayon::iter::plumbing::bridge_producer_consumer::helper::_$u7b$$u7b$closure$u7d$$u7d$::hea893d99d2794791 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:427:21
#26 0x7fe1b93117c9 in rayon_core::join::join_context::call_b::_$u7b$$u7b$closure$u7d$$u7d$::h49bf5a792680eaa9 src/third_party/rust/rayon-core/src/join/mod.rs:129:25
#27 0x7fe1b93117c9 in rayon_core::job::StackJob$LT$L$C$F$C$R$GT$::run_inline::h1e263ba69f795e5f src/third_party/rust/rayon-core/src/job.rs:97:9
#28 0x7fe1b93117c9 in rayon_core::join::join_context::_$u7b$$u7b$closure$u7d$$u7d$::h20cb11dcd0d2a430 src/third_party/rust/rayon-core/src/join/mod.rs:158:36
#29 0x7fe1b9360b56 in rayon_core::registry::in_worker::h7ef39ba38031c680 src/third_party/rust/rayon-core/src/registry.rs:879:13
#30 0x7fe1b9360b56 in rayon_core::join::join_context::h84f142bc3fc0819c src/third_party/rust/rayon-core/src/join/mod.rs:132:5
#31 0x7fe1b9360b56 in rayon::iter::plumbing::bridge_producer_consumer::helper::hc4d3c979010b9aa4 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:416:47
#32 0x7fe1b93117c9 in rayon::iter::plumbing::bridge_producer_consumer::helper::_$u7b$$u7b$closure$u7d$$u7d$::hea893d99d2794791 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:427:21
#33 0x7fe1b93117c9 in rayon_core::join::join_context::call_b::_$u7b$$u7b$closure$u7d$$u7d$::h49bf5a792680eaa9 src/third_party/rust/rayon-core/src/join/mod.rs:129:25
#34 0x7fe1b93117c9 in rayon_core::job::StackJob$LT$L$C$F$C$R$GT$::run_inline::h1e263ba69f795e5f src/third_party/rust/rayon-core/src/job.rs:97:9
#35 0x7fe1b93117c9 in rayon_core::join::join_context::_$u7b$$u7b$closure$u7d$$u7d$::h20cb11dcd0d2a430 src/third_party/rust/rayon-core/src/join/mod.rs:158:36
#36 0x7fe1b9360b56 in rayon_core::registry::in_worker::h7ef39ba38031c680 src/third_party/rust/rayon-core/src/registry.rs:879:13
#37 0x7fe1b9360b56 in rayon_core::join::join_context::h84f142bc3fc0819c src/third_party/rust/rayon-core/src/join/mod.rs:132:5
#38 0x7fe1b9360b56 in rayon::iter::plumbing::bridge_producer_consumer::helper::hc4d3c979010b9aa4 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:416:47
#39 0x7fe1b931164b in rayon::iter::plumbing::bridge_producer_consumer::helper::_$u7b$$u7b$closure$u7d$$u7d$::h995f010e6bd84cf1 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:418:21
#40 0x7fe1b931164b in rayon_core::join::join_context::call_a::_$u7b$$u7b$closure$u7d$$u7d$::h8afcaf31a6d23c85 src/third_party/rust/rayon-core/src/join/mod.rs:124:17
#41 0x7fe1b931164b in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::hccfe90cd1f3d484c /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panic.rs:322:9
#42 0x7fe1b931164b in std::panicking::try::do_call::habd93a99954570c6 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panicking.rs:379:40
#43 0x7fe1b931164b in std::panicking::try::h49a3212107ddbe03 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panicking.rs:343:19
#44 0x7fe1b931164b in std::panic::catch_unwind::h9befc5b398df7055 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panic.rs:396:14
#45 0x7fe1b931164b in rayon_core::unwind::halt_unwinding::h2d8eb3f3aab72554 src/third_party/rust/rayon-core/src/unwind.rs:17:5
#46 0x7fe1b931164b in rayon_core::join::join_context::_$u7b$$u7b$closure$u7d$$u7d$::h20cb11dcd0d2a430 src/third_party/rust/rayon-core/src/join/mod.rs:141:24
#47 0x7fe1b9360b56 in rayon_core::registry::in_worker::h7ef39ba38031c680 src/third_party/rust/rayon-core/src/registry.rs:879:13
#48 0x7fe1b9360b56 in rayon_core::join::join_context::h84f142bc3fc0819c src/third_party/rust/rayon-core/src/join/mod.rs:132:5
#49 0x7fe1b9360b56 in rayon::iter::plumbing::bridge_producer_consumer::helper::hc4d3c979010b9aa4 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:416:47
#50 0x7fe1b9363f37 in rayon::iter::plumbing::bridge_producer_consumer::helper::_$u7b$$u7b$closure$u7d$$u7d$::hea893d99d2794791 src/third_party/rust/rayon/src/iter/plumbing/mod.rs:427:21
#51 0x7fe1b9363f37 in rayon_core::join::join_context::call_b::_$u7b$$u7b$closure$u7d$$u7d$::h49bf5a792680eaa9 src/third_party/rust/rayon-core/src/join/mod.rs:129:25
#52 0x7fe1b9363f37 in _$LT$rayon_core..job..StackJob$LT$L$C$F$C$R$GT$$u20$as$u20$rayon_core..job..Job$GT$::execute::call::_$u7b$$u7b$closure$u7d$$u7d$::h9a062ed40fe087ff src/third_party/rust/rayon-core/src/job.rs:113:21
#53 0x7fe1b9363f37 in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h9b902b9b9c829719 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panic.rs:322:9
#54 0x7fe1b9363f37 in std::panicking::try::do_call::h65af5162d6d970a8 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panicking.rs:379:40
#55 0x7fe1b9363f37 in std::panicking::try::hb28b46c285e88265 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panicking.rs:343:19
#56 0x7fe1b9363f37 in std::panic::catch_unwind::h2853c1be63cb260b /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panic.rs:396:14
#57 0x7fe1b9363f37 in rayon_core::unwind::halt_unwinding::h51b5e8456a6b0a8a src/third_party/rust/rayon-core/src/unwind.rs:17:5
#58 0x7fe1b9363f37 in _$LT$rayon_core..job..StackJob$LT$L$C$F$C$R$GT$$u20$as$u20$rayon_core..job..Job$GT$::execute::hf9c50a8ffd6e6766 src/third_party/rust/rayon-core/src/job.rs:119:38
#59 0x7fe1bad0bac0 in rayon_core::job::JobRef::execute::h6f62ba254fb9d1ac src/third_party/rust/rayon-core/src/job.rs:59:9
#60 0x7fe1bad0bac0 in rayon_core::registry::WorkerThread::execute::h9069da8ae89c6e04 src/third_party/rust/rayon-core/src/registry.rs:753:9
#61 0x7fe1bad0bac0 in rayon_core::registry::WorkerThread::wait_until_cold::hbee764ad73856a22 src/third_party/rust/rayon-core/src/registry.rs:730:17
#62 0x7fe1bad08077 in rayon_core::registry::WorkerThread::wait_until::h54bfb27a674b3541 src/third_party/rust/rayon-core/src/registry.rs:704:13
#63 0x7fe1bad08077 in rayon_core::registry::main_loop::h01723bb8a2cc19f0 src/third_party/rust/rayon-core/src/registry.rs:837:5
#64 0x7fe1bad08077 in rayon_core::registry::ThreadBuilder::run::hcdd1536eb6e23685 src/third_party/rust/rayon-core/src/registry.rs:56:18
#65 0x7fe1bacfb23b in _$LT$rayon_core..registry..DefaultSpawn$u20$as$u20$rayon_core..registry..ThreadSpawn$GT$::spawn::_$u7b$$u7b$closure$u7d$$u7d$::h7c9a1322097f6654 src/third_party/rust/rayon-core/src/registry.rs:101:20
#66 0x7fe1bacfb23b in std::sys_common::backtrace::__rust_begin_short_backtrace::he9c2fb7c77308e33 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/sys_common/backtrace.rs:125:18
#67 0x7fe1bacfd66a in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h2d56b244e2776e48 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/thread/mod.rs:474:17
#68 0x7fe1bacfd66a in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::h644e4b3e80849393 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panic.rs:322:9
#69 0x7fe1bacfd66a in std::panicking::try::do_call::ha5395542df26cd0d /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panicking.rs:379:40
#70 0x7fe1bacfd66a in std::panicking::try::h9674f97365e02ce2 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panicking.rs:343:19
#71 0x7fe1bacfd66a in std::panic::catch_unwind::h5d746f403b4a6019 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/panic.rs:396:14
#72 0x7fe1bacfd66a in std::thread::Builder::spawn_unchecked::_$u7b$$u7b$closure$u7d$$u7d$::h08d8f35d38e14cb8 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/std/src/thread/mod.rs:473:30
#73 0x7fe1bacfd66a in core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::hf8686da4dedc8d51 /builds/worker/fetches/rustc/lib/rustlib/src/rust/library/core/src/ops/function.rs:227:5
#74 0x7fe1bae154f9 in _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::h9ed215ba67984d70 /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/alloc/src/boxed.rs:1328:9
#75 0x7fe1bae154f9 in _$LT$alloc..boxed..Box$LT$F$C$A$GT$$u20$as$u20$core..ops..function..FnOnce$LT$Args$GT$$GT$::call_once::hcece06e1fe04906f /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/alloc/src/boxed.rs:1328:9
#76 0x7fe1bae154f9 in std::sys::unix::thread::Thread::new::thread_start::h6e82a4b7be15319a /rustc/cb75ad5db02783e8b0222fee363c5f63f7e2cf5b/library/std/src/sys/unix/thread.rs:71:17
#77 0x7fe1c6d4a608 in start_thread /build/glibc-eX1tMB/glibc-2.31/nptl/pthread_create.c:477:8
#78 0x7fe1c6913292 in clone /build/glibc-eX1tMB/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Comment 1

[Tyson Smith [:tsmith]]

Attachment: prefs.js

Comment 2

[Bugmon [:jkratzer for issues]]

Bugmon Analysis:
Verified bug as reproducible on mozilla-central 20210323214359-ea38e210ba4d.
The bug appears to have been introduced in the following build range:

Start: 31f2b92312abea75502d1e1d4eb4ccf9225f740d (20201128213548)
End: b0865ea584621ce9e7f68833565e3d8ae117ce32 (20201130093031)
Pushlog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=31f2b92312abea75502d1e1d4eb4ccf9225f740d&tochange=b0865ea584621ce9e7f68833565e3d8ae117ce32

Comment 3

[Andrew Osmond [:aosmond] (he/him)]

The regression window is because of bug 1677857 which added some new fuzzing methods; the most relevant one being the memory pressure event used in the test page. So the problem likely existed prior to this, we just had no means to reliably reproduce it. Memory pressure probably causes us to discard something prematurely, and the font handles get cleared before we finished processing the blob image.

Comment 4

[Andrew Osmond [:aosmond] (he/him)]

One of these arrays gets cleared during the memory pressure event:

https://searchfox.org/mozilla-central/rev/bb37e6fe8bbe383a57a4ad21a201e26416613ca1/gfx/layers/wr/WebRenderUserData.h#223
https://searchfox.org/mozilla-central/rev/bb37e6fe8bbe383a57a4ad21a201e26416613ca1/gfx/layers/wr/WebRenderCommandBuilder.cpp#319
https://searchfox.org/mozilla-central/rev/bb37e6fe8bbe383a57a4ad21a201e26416613ca1/gfx/layers/wr/WebRenderCommandBuilder.cpp#2462

Which in turn causes a ResourceUpdate::DeleteFontInstance eventually. The blobs block the scene builder thread, but we delete the font instance on the compositor thread when we send the transaction:

https://searchfox.org/mozilla-central/rev/bb37e6fe8bbe383a57a4ad21a201e26416613ca1/gfx/wr/webrender/src/api_resources.rs#157

We shouldn't delete font instances from Moz2DImageRenderer::sBlobFontTable until after the blobs are completed, e.g. do it on the scene builder thread.

Comment 5

[Andrew Osmond [:aosmond] (he/him)]

I suspect we see this in the wild under the signature wr_moz2d_render_cb, e.g.:

https://crash-stats.mozilla.org/report/index/35b86745-ff20-4cc9-8927-170ce0210325
https://crash-stats.mozilla.org/report/index/fe1056d1-05e4-4ad4-b4cd-08efd0210324

https://crash-stats.mozilla.org/signature/?proto_signature=~wr_moz2d_render_cb&moz_crash_reason=~MOZ_CRASH%28GFX_CRASH%29&signature=wr_moz2d_render_cb&date=%3E%3D2021-03-11T19%3A29%3A00.000Z&date=%3C2021-03-25T19%3A29%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#reports

Comment 6

[Tyson Smith [:tsmith]]

A Pernosco session is available here: https://pernos.co/debug/VYliXUyJ7Ht77jxrg0uVkQ/index.html

Comment 7

[Intermittent Failures Robot]

2 failures in 2580 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• macosx1015-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1700494&startday=2021-08-23&endday=2021-08-29&tree=all

Comment 8

[Intermittent Failures Robot]

1 failures in 3084 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1700494&startday=2021-08-30&endday=2021-09-05&tree=all

Comment 9

[Bugmon [:jkratzer for issues]]

Bugmon Analysis
Testcase crashes using the initial build (mozilla-central 20210121213347-e5e734f4fc19) but not with tip (mozilla-central 20211224215233-818c851d0aeb.)
The bug appears to have been fixed in the following build range:

Start: d1c894f81d2a11efc998f4294fe137cb371c1d2b (20211213201156)
End: 51773d1ab7b50883b4c8ca2322bd646f122ece3e (20211214042524)
Pushlog: https://hg.mozilla.org/mozilla-unified/pushloghtml?fromchange=d1c894f81d2a11efc998f4294fe137cb371c1d2b&tochange=51773d1ab7b50883b4c8ca2322bd646f122ece3e
Removing bugmon keyword as no further action possible. Please review the bug and re-add the keyword for further analysis.