1511493 - assertion failed: self.font_contexts.lock_shared_context().has_font(&font.font_key)

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.html

==13678==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7f9f9a5d25ae bp 0x7f9f242b7f40 sp 0x7f9f242b7f40 T110)
==13678==The signal is caused by a WRITE memory access.
==13678==Hint: address points to the zero page.
    #0 0x7f9f9a5d25ad in MOZ_CrashOOL(char const*, int, char const*) src/obj-firefox/dist/include/mozilla/Assertions.h:311:3
    #1 0x7f9f9a5d256a in GeckoCrashOOL src/toolkit/xre/nsAppRunner.cpp:5349:3
    #2 0x7f9f9c54835a in gkrust_shared::panic_hook::h577176513f96817f src/toolkit/library/rust/shared/lib.rs:234:8
    #3 0x7f9f9c548298 in core::ops::function::Fn::call::h82a5285a736af5e0 src/libcore/ops/function.rs:78:4
    #4 0x7f9f9cc94368 in std::panicking::rust_panic_with_hook::h0e12cb2fc86d00fa /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libstd/panicking.rs:481:16
    #5 0x7f9f9cc9415d in std::panicking::continue_panic_fmt::h141671b29fe0e27d /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libstd/panicking.rs:391:4
    #6 0x7f9f9cc96365 in rust_begin_unwind /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libstd/panicking.rs:326:4
    #7 0x7f9f9cca7ccb in core::panicking::panic_fmt::h429a06507aba9228 /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libcore/panicking.rs:77:13
    #8 0x7f9f9cca7c8a in core::panicking::panic::haa57ffd51eb03b56 /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libcore/panicking.rs:52:4
    #9 0x7f9f9c864963 in webrender::picture::PicturePrimitive::prepare_for_render::h24037f301e8a4fad src/gfx/wr/webrender/src/freelist.rs
    #10 0x7f9f9c8416ff in webrender::prim_store::PrimitiveStore::prepare_prim_for_render::h360860d9989ed1a3 src/gfx/wr/webrender/src/prim_store.rs:3068:19
    #11 0x7f9f9c827fb5 in webrender::prim_store::PrimitiveStore::prepare_primitives::hb6da2a4df37f75e8 src/gfx/wr/webrender/src/prim_store.rs:3204:15
    #12 0x7f9f9c841126 in webrender::prim_store::PrimitiveStore::prepare_prim_for_render::h360860d9989ed1a3 src/gfx/wr/webrender/src/prim_store.rs:2871:16
    #13 0x7f9f9c827fb5 in webrender::prim_store::PrimitiveStore::prepare_primitives::hb6da2a4df37f75e8 src/gfx/wr/webrender/src/prim_store.rs:3204:15
    #14 0x7f9f9c841126 in webrender::prim_store::PrimitiveStore::prepare_prim_for_render::h360860d9989ed1a3 src/gfx/wr/webrender/src/prim_store.rs:2871:16
    #15 0x7f9f9c827fb5 in webrender::prim_store::PrimitiveStore::prepare_primitives::hb6da2a4df37f75e8 src/gfx/wr/webrender/src/prim_store.rs:3204:15
    #16 0x7f9f9c841126 in webrender::prim_store::PrimitiveStore::prepare_prim_for_render::h360860d9989ed1a3 src/gfx/wr/webrender/src/prim_store.rs:2871:16
    #17 0x7f9f9c827fb5 in webrender::prim_store::PrimitiveStore::prepare_primitives::hb6da2a4df37f75e8 src/gfx/wr/webrender/src/prim_store.rs:3204:15
    #18 0x7f9f9c841126 in webrender::prim_store::PrimitiveStore::prepare_prim_for_render::h360860d9989ed1a3 src/gfx/wr/webrender/src/prim_store.rs:2871:16
    #19 0x7f9f9c827fb5 in webrender::prim_store::PrimitiveStore::prepare_primitives::hb6da2a4df37f75e8 src/gfx/wr/webrender/src/prim_store.rs:3204:15
    #20 0x7f9f9c841126 in webrender::prim_store::PrimitiveStore::prepare_prim_for_render::h360860d9989ed1a3 src/gfx/wr/webrender/src/prim_store.rs:2871:16
    #21 0x7f9f9c827fb5 in webrender::prim_store::PrimitiveStore::prepare_primitives::hb6da2a4df37f75e8 src/gfx/wr/webrender/src/prim_store.rs:3204:15
    #22 0x7f9f9c813d02 in webrender::frame_builder::FrameBuilder::build_layer_screen_rects_and_cull_layers::hbae8f73d8f1284ab src/gfx/wr/webrender/src/frame_builder.rs:315:8
    #23 0x7f9f9c813d02 in webrender::frame_builder::FrameBuilder::build::h82c647da6420921e src/gfx/wr/webrender/src/frame_builder.rs:401
    #24 0x7f9f9c7f4b34 in webrender::render_backend::Document::build_frame::h3f2fd0e88f6987ca src/gfx/wr/webrender/src/render_backend.rs:418:24
    #25 0x7f9f9c7cffb2 in webrender::render_backend::RenderBackend::update_document::h74cbceccd5d3ce5a src/gfx/wr/webrender/src/render_backend.rs:1239:40
    #26 0x7f9f9c7dc4b4 in webrender::render_backend::RenderBackend::prepare_transaction::h2c9450f08d84097f src/gfx/wr/webrender/src/render_backend.rs:1114:12
    #27 0x7f9f9c7dc4b4 in webrender::render_backend::RenderBackend::process_api_msg::h7d9d58e692c914ce src/gfx/wr/webrender/src/render_backend.rs:1052
    #28 0x7f9f9c7bd719 in webrender::render_backend::RenderBackend::run::hd82519aa9035be53 src/gfx/wr/webrender/src/render_backend.rs:827:20
    #29 0x7f9f9c7b8be4 in webrender::renderer::Renderer::new::_$u7b$$u7b$closure$u7d$$u7d$::hb49c225a8fd7fa5a src/gfx/wr/webrender/src/renderer.rs:1952:12
    #30 0x7f9f9c7b8be4 in std::sys_common::backtrace::__rust_begin_short_backtrace::h47168340fd24836a src/libstd/sys_common/backtrace.rs:136
    #31 0x7f9f9c7b7cfb in std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::haedab32c738a89d5 src/libstd/thread/mod.rs:409:20
    #32 0x7f9f9c7b7cfb in _$LT$std..panic..AssertUnwindSafe$LT$F$GT$$u20$as$u20$core..ops..function..FnOnce$LT$$LP$$RP$$GT$$GT$::call_once::ha2c6a7af3babcf9d src/libstd/panic.rs:313
    #33 0x7f9f9c7b7cfb in std::panicking::try::do_call::h8d4a148903e51c01 src/libstd/panicking.rs:310
    #34 0x7f9f9c7b7cfb in __rust_maybe_catch_panic /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libpanic_abort/lib.rs:41
    #35 0x7f9f9c7b7cfb in std::panicking::try::hf5e2cdd67127d94a src/libstd/panicking.rs:289
    #36 0x7f9f9c7b7cfb in std::panic::catch_unwind::h8f0edd70beda7774 src/libstd/panic.rs:392
    #37 0x7f9f9c7b7cfb in std::thread::Builder::spawn::_$u7b$$u7b$closure$u7d$$u7d$::ha8a701be54d9a00d src/libstd/thread/mod.rs:408
    #38 0x7f9f9c7b7cfb in _$LT$F$u20$as$u20$alloc..boxed..FnBox$LT$A$GT$$GT$::call_box::h30e70ed6e86cc7a1 src/liballoc/boxed.rs:646
    #39 0x7f9f9cc96941 in _$LT$alloc..boxed..Box$LT$$LP$dyn$u20$alloc..boxed..FnBox$LT$A$C$$u20$Output$u3d$R$GT$$u20$$u2b$$u20$$u27$a$RP$$GT$$u20$as$u20$core..ops..function..FnOnce$LT$A$GT$$GT$::call_once::hd022a0500f5eee13 /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/liballoc/boxed.rs:656:8
    #40 0x7f9f9cc96941 in std::sys_common::thread::start_thread::hb77935aee02382da /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libstd/sys_common/thread.rs:24
    #41 0x7f9f9cc96941 in std::sys::unix::thread::Thread::new::thread_start::hd61f7429b3de1d75 /rustc/da5f414c2c0bfe5198934493f04c676e2b23ff2e/src/libstd/sys/unix/thread.rs:90
    #42 0x7f9fb07686b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
    #43 0x7f9faf7f141c in clone /build/glibc-Cl5G7W/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109

Thread T93 (Renderer) created by T0 here:
    #0 0x5607e9a2f6ad in __interceptor_pthread_create /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:210:3
    #1 0x7f9f8caf9a72 in CreateThread src/ipc/chromium/src/base/platform_thread_posix.cc:127:14
    #2 0x7f9f8caf9a72 in PlatformThread::Create(unsigned long, PlatformThread::Delegate*, unsigned long*) src/ipc/chromium/src/base/platform_thread_posix.cc:138
    #3 0x7f9f8cb2ff1f in base::Thread::StartWithOptions(base::Thread::Options const&) src/ipc/chromium/src/base/thread.cc:102:8
    #4 0x7f9f8f482b36 in mozilla::wr::RenderThread::Start() src/gfx/webrender_bindings/RenderThread.cpp:73:16
    #5 0x7f9f8f145616 in gfxPlatform::InitLayersIPC() src/gfx/thebes/gfxPlatform.cpp:1294:7
    #6 0x7f9f8f13db07 in gfxPlatform::Init() src/gfx/thebes/gfxPlatform.cpp:996:5
    #7 0x7f9f8f13aec3 in gfxPlatform::GetPlatform() src/gfx/thebes/gfxPlatform.cpp:523:9
    #8 0x7f9f95b1ddb8 in mozilla::widget::GfxInfoBase::GetContentBackend(nsTSubstring<char16_t>&) src/widget/GfxInfoBase.cpp:1522:25
    #9 0x7f9f8b9a7811 in NS_InvokeByIndex src/xpcom/reflect/xptcall/md/unix/xptcinvoke_asm_x86_64_unix.S:106
    #10 0x7f9f8dccf787 in Invoke src/js/xpconnect/src/XPCWrappedNative.cpp:1736:12
    #11 0x7f9f8dccf787 in Call src/js/xpconnect/src/XPCWrappedNative.cpp:1269
    #12 0x7f9f8dccf787 in XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode) src/js/xpconnect/src/XPCWrappedNative.cpp:1233
    #13 0x7f9f8dcd81cf in GetAttribute src/js/xpconnect/src/xpcprivate.h:1589:17
    #14 0x7f9f8dcd81cf in XPC_WN_GetterSetter(JSContext*, unsigned int, JS::Value*) src/js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1061
    #15 0x7f9f9c3f348d in CallJSNative src/js/src/vm/Interpreter.cpp:468:15
    #16 0x7f9f9c3f348d in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct) src/js/src/vm/Interpreter.cpp:560
    #17 0x7f9f9c3f7db0 in InternalCall src/js/src/vm/Interpreter.cpp:614:12
    #18 0x7f9f9c3f7db0 in Call src/js/src/vm/Interpreter.cpp:634
    #19 0x7f9f9c3f7db0 in js::CallGetter(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, JS::MutableHandle<JS::Value>) src/js/src/vm/Interpreter.cpp:761
    #20 0x7f9f9ade3579 in CallGetter src/js/src/vm/NativeObject.cpp:2304:16
    #21 0x7f9f9ade3579 in GetExistingProperty<js::CanGC> src/js/src/vm/NativeObject.cpp:2359
    #22 0x7f9f9ade3579 in NativeGetPropertyInline<js::CanGC> src/js/src/vm/NativeObject.cpp:2616
    #23 0x7f9f9ade3579 in js::NativeGetProperty(JSContext*, JS::Handle<js::NativeObject*>, JS::Handle<JS::Value>, JS::Handle<JS::PropertyKey>, JS::MutableHandle<JS::Value>) src/js/src/vm/NativeObject.cpp:2653
    #24 0x7f9f9c3db963 in GetProperty src/js/src/vm/ObjectOperations-inl.h:122:12
    #25 0x7f9f9c3db963 in GetObjectElementOperation src/js/src/vm/Interpreter-inl.h:558
    #26 0x7f9f9c3db963 in GetElementOperation src/js/src/vm/Interpreter-inl.h:675
    #27 0x7f9f9c3db963 in Interpret(JSContext*, js::RunState&) src/js/src/vm/Interpreter.cpp:3315
    #28 0x7f9f9c3c06d6 in js::RunScript(JSContext*, js::RunState&) src/js/src/vm/Interpreter.cpp:447:12
    #29 0x7f9f9c3f3e31 in js::InternalCallOrConstruct(JSContext*, JS::CallArgs const&, js::MaybeConstruct) src/js/src/vm/Interpreter.cpp:587:15
    #30 0x7f9f9c3f5ab2 in js::Call(JSContext*, JS::Handle<JS::Value>, JS::Handle<JS::Value>, js::AnyInvokeArgs const&, JS::MutableHandle<JS::Value>) src/js/src/vm/Interpreter.cpp:634:10
    #31 0x7f9f9b3b5b4a in JS_CallFunctionValue(JSContext*, JS::Handle<JSObject*>, JS::Handle<JS::Value>, JS::HandleValueArray const&, JS::MutableHandle<JS::Value>) src/js/src/jsapi.cpp:2932:12
    #32 0x7f9f8dcb1929 in nsXPCWrappedJSClass::CallMethod(nsXPCWrappedJS*, unsigned short, nsXPTMethodInfo const*, nsXPTCMiniVariant*) src/js/xpconnect/src/XPCWrappedJSClass.cpp:1169:23
    #33 0x7f9f8b9a8f18 in PrepareAndDispatch src/xpcom/reflect/xptcall/md/unix/xptcstubs_x86_64_linux.cpp:127:37
    #34 0x7f9f8b9a7dea in SharedStub (libxul.so+0x4910dea)
    #35 0x7f9f8b8f3f2d in NS_CreateServicesFromCategory(char const*, nsISupports*, char const*, char16_t const*) src/xpcom/components/nsCategoryManager.cpp:777:19
    #36 0x7f9f9a5f7723 in nsXREDirProvider::DoStartup() src/toolkit/xre/nsXREDirProvider.cpp:1103:11
    #37 0x7f9f9a5cc467 in XREMain::XRE_mainRun() src/toolkit/xre/nsAppRunner.cpp:4620:16
    #38 0x7f9f9a5cfe19 in XREMain::XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:4935:8
    #39 0x7f9f9a5d18e3 in XRE_main(int, char**, mozilla::BootstrapConfig const&) src/toolkit/xre/nsAppRunner.cpp:5027:21
    #40 0x5607e9a7967c in do_main src/browser/app/nsBrowserApp.cpp:233:22
    #41 0x5607e9a7967c in main src/browser/app/nsBrowserApp.cpp:315
    #42 0x7f9faf70a82f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291

Comment 1

[Tyson Smith [:tsmith]]

Attachment: testcase.html

Updated test case.

Comment 2

[Tyson Smith [:tsmith]]

Please let me know if a Pernosco session would be helpful and I will create one.

Comment 3

[Darkspirit]

bp-0a1062a5-608d-4b04-a915-bdcad0191024

Comment 5

[Takanori MATSUURA]

bp-e1c49f4a-3d46-434a-bfaf-3908d0200524

Comment 6

[Julien Cristau [:jcristau]]

The crash spike with this signature in nightly was fixed in bug 1640401.

Comment 7

[Tyson Smith [:tsmith]]

The attached test case no longer reproduces this issue.

Comment 8

[John]

Just crashed with 0b5c6ef1-b3f3-4928-91bd-98ed10200802

Comment 11

[Darkspirit]

Some signatures with this crash reason even mention DejaVuSans.ttf, NotoSansCJK-Bold.ttc, NotoSansCJK-Regular.ttc, NotoSansCJK-Thin.ttc.

Comment 12

[Nicolas Silva [:nical]]

Fonts are added asynchronously to the shared context via async_for_each 1. I don't see a mechanism to ensure that this it will happen before the frame_building code starts requesting glyphs. So it could be a race condition.

One way to work around it is (A) to change async_for_each to be a bit less async and do the shared_context work on the calling thread. We'd pay the overhead of adding/deleting fonts on the render backend thread but at least add_font would be guaranteed proper ordering with respect to request_glyph.
The other thing we could do (B) is to not have this assertion in request_glyph, and panic when rasterizing the glyph instead. It would not add any overhead but if the bug isn't the race condition I described, then it will move the signature which gets in the way of tracking and solving it.

I propose that we first do (A), at the risk of taking a perf hit and see if it fixes the crash. If it does and the regression is worse than what we're comfortable with, then we can do (B).

Comment 13

[Nicolas Silva [:nical]]

Actually I think that we are protected by blocking the calling thread until all font contexts have been locked in https://searchfox.org/mozilla-central/rev/27932d4e6ebd2f4b8519865dad864c72176e4e3b/gfx/wr/webrender/src/glyph_rasterizer/mod.rs#870

So that would rule out the race condition I mentioned earlier.

Comment 14

[Nicolas Silva [:nical]]

Next idea: If a font fails sanitization, WebRender doesn't know about it. Perhaps we should have a special placeholder added for fonts that faild sanitization, if only to have more context when failing to request the glyphs.

Comment 15

[Wayne Mery (:wsmwk)]

Seeing this with Thunderbird. All linux https://crash-stats.mozilla.org/signature/?product=Thunderbird&signature=webrender%3A%3Aprepare%3A%3Aprepare_interned_prim_for_render&date=%3E%3D2021-06-11T08%3A43%3A00.000Z&date=%3C2021-09-11T08%3A43%3A00.000Z

Comment 16

[Lee Salzman [:lsalzman]]

Attachment: Bug 1511493 - Ensure PushGlyphs uses the current transaction's IpcResourceUpdateQueue. r?nical

WebRenderBridgeChild::GetFontKeyForScaledFont can currently cause a IpcResourceUpdateQueue race.
If we're in the middle of a transaction building a blob image, GetFontKeyForScaledFont is called
in the blob image building code using the transaction's IpcResourceUpdateQueue as expected, such
that resource updates are sent out when the transaction is finalized.

However, TextDrawTarget calls into PushGlyphs without passing along its IpcResourceUpdateQueue,
calling GetFontKeyForScaledFont without it, and causing it to immediately send out the resource
update.

So if a blob image uses a font key and submits a resource update, but a display list is built
after that also using the font key within the transaction, the display list will fail to send
the resource update because it thinks the blob image already did, even though the blob image
transaction has not yet been finalized.

The simple fix is to just pass IpcResourceUpdateQueue from TextDrawTarget into PushGlyphs, thus
ensuring the resource updates are properly ordered.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e6dde5d8fdba
Ensure PushGlyphs uses the current transaction's IpcResourceUpdateQueue. r=emilio

Comment 18

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/e6dde5d8fdba

Comment 19

[Alex Finder]

== Change summary for alert #33534 (as of Tue, 15 Mar 2022 13:11:18 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
7%	linkedin ContentfulSpeedIndex	macosx1015-64-shippable-qr	cold fission webrender	1,930.79 -> 1,803.75
6%	linkedin ContentfulSpeedIndex	macosx1015-64-shippable-qr	cold fission webrender	1,913.67 -> 1,806.00
4%	google-slides ContentfulSpeedIndex	linux1804-64-shippable-qr	cold fission webrender	1,257.96 -> 1,212.67

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=33534