Closed Bug 1827340 Opened 2 years ago Closed 2 years ago

crash in [@ blendTextureNearestFast]

Categories

(Core :: Graphics: WebRender, defect)

defect

Tracking

()

VERIFIED FIXED
115 Branch
Tracking Status
firefox-esr102 114+ fixed
firefox112 --- unaffected
firefox113 --- wontfix
firefox114 + fixed
firefox115 + fixed

People

(Reporter: tsmith, Unassigned)

References

(Blocks 1 open bug, Regression)

Details

(6 keywords, Whiteboard: [bugmon:bisected,confirmed][adv-main114+r][adv-esr102.12+r])

Crash Data

Attachments

(6 files)

Attached file testcase.zip

Found while fuzzing m-c 20230406-335d0840e3dd (--enable-address-sanitizer --enable-fuzzing)

This issue seems to only be reliable in some cases. It reproduces on one of three machines I've tested it on (Ubuntu 22.04).

A Pernosco session is available here: https://pernos.co/debug/5SqcLzX8O2oewUhSEA574Q/index.html

To reproduce via Grizzly Replay:

$ pip install fuzzfetch grizzly-framework
$ python -m fuzzfetch -a --fuzzing -n firefox
$ python -m grizzly.replay ./firefox/firefox testcase.zip --xvfb --repeat 10 --no-harness

This test is packed in a zip file to help reproduction via bugmon. Only testcase.html contained in the zip is required to reproduce the issue.

==93023==ERROR: AddressSanitizer: SEGV on unknown address 0x7fe2d12e1000 (pc 0x7fe30d26f122 bp 0x7fe2d908d590 sp 0x7fe2d908d0a0 T27)
==93023==The signal is caused by a READ memory access.
    #0 0x7fe30d26f122 in load<unsigned int> /gecko/gfx/wr/swgl/src/vector_type.h:503:5
    #1 0x7fe30d26f122 in unaligned_load<unsigned char __attribute__((ext_vector_type(16))), unsigned int> /gecko/gfx/wr/swgl/src/vector_type.h:532:10
    #2 0x7fe30d26f122 in int blendTextureNearestFast<true, glsl::sampler2DRect_impl*, NoColor, unsigned int>(glsl::sampler2DRect_impl*, glsl::vec2, int, glsl::vec4_scalar const&, NoColor, unsigned int*) /gecko/gfx/wr/swgl/src/swgl_ext.h:517:27
    #3 0x7fe30d2d8431 in brush_image_ALPHA_PASS_TEXTURE_RECT_frag::swgl_drawSpanRGBA8() /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/build/swgl-4f0dca493aca307b/out/brush_image_ALPHA_PASS_TEXTURE_RECT.h:885:2
    #4 0x7fe30d2cf532 in brush_image_ALPHA_PASS_TEXTURE_RECT_frag::draw_span_RGBA8(glsl::FragmentShaderImpl*) /builds/worker/workspace/obj-build/x86_64-unknown-linux-gnu/release/build/swgl-4f0dca493aca307b/out/brush_image_ALPHA_PASS_TEXTURE_RECT.h:933:28
    #5 0x7fe30d5ce2ba in draw_span /gecko/gfx/wr/swgl/src/program.h:168:12
    #6 0x7fe30d5ce2ba in void draw_quad_spans<unsigned int>(int, glsl::vec2_scalar*, unsigned int, glsl::vec3*, Texture&, Texture&, ClipRect const&) /gecko/gfx/wr/swgl/src/rasterize.h:1034:42
    #7 0x7fe30d0d1d13 in draw_quad(int, Texture&, Texture&) /gecko/gfx/wr/swgl/src/rasterize.h:1625:5
    #8 0x7fe30d0cdbf6 in void draw_elements<unsigned short>(int, int, unsigned long, VertexArray&, Texture&, Texture&) /gecko/gfx/wr/swgl/src/rasterize.h:1655:5
    #9 0x7fe30d0cd83e in DrawElementsInstanced /gecko/gfx/wr/swgl/src/gl.cc:2748:7
    #10 0x7fe30c94c848 in webrender::device::gl::Device::draw_indexed_triangles_instanced_u16::hbad93f51dbf0e657 /gecko/gfx/wr/webrender/src/device/gl.rs:3723:9
    #11 0x7fe30c94c848 in webrender::renderer::Renderer::draw_instanced_batch::hbc7db5b58bf17bd7 /gecko/gfx/wr/webrender/src/renderer/mod.rs:2018:17
    #12 0x7fe30c95ccbb in webrender::renderer::Renderer::draw_alpha_batch_container::h15d61c1f6a1bd80e /gecko/gfx/wr/webrender/src/renderer/mod.rs:2673:17
    #13 0x7fe30c99f8e4 in webrender::renderer::Renderer::draw_color_target::h19eb028f497936ee /gecko/gfx/wr/webrender/src/renderer/mod.rs:3378:13
    #14 0x7fe30c99f8e4 in webrender::renderer::Renderer::draw_frame::hc06fd927699f4fff /gecko/gfx/wr/webrender/src/renderer/mod.rs:4520:17
    #15 0x7fe30c90bfce in webrender::renderer::Renderer::render_impl::h9593e87977ebeccb /gecko/gfx/wr/webrender/src/renderer/mod.rs:1514:17
    #16 0x7fe30c906876 in webrender::renderer::Renderer::render::he6edbb5e898fd9bb /gecko/gfx/wr/webrender/src/renderer/mod.rs:1231:30
    #17 0x7fe30bc703e5 in wr_renderer_render /gecko/gfx/webrender_bindings/src/bindings.rs:619:11
    #18 0x7fe2fca4ba95 in mozilla::wr::RendererOGL::UpdateAndRender(mozilla::Maybe<mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits>> const&, mozilla::Maybe<mozilla::wr::ImageFormat> const&, mozilla::Maybe<mozilla::Range<unsigned char>> const&, bool*, mozilla::wr::RendererStats*) /gecko/gfx/webrender_bindings/RendererOGL.cpp:190:19
    #19 0x7fe2fca49f1d in mozilla::wr::RenderThread::UpdateAndRender(mozilla::wr::WrWindowId, mozilla::layers::BaseTransactionId<mozilla::VsyncIdType> const&, mozilla::TimeStamp const&, bool, mozilla::Maybe<mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits>> const&, mozilla::Maybe<mozilla::wr::ImageFormat> const&, mozilla::Maybe<mozilla::Range<unsigned char>> const&, bool*) /gecko/gfx/webrender_bindings/RenderThread.cpp:849:31
    #20 0x7fe2fca48edd in mozilla::wr::RenderThread::HandleFrameOneDocInner(mozilla::wr::WrWindowId, bool, bool, mozilla::Maybe<mozilla::wr::FramePublishId>) /gecko/gfx/webrender_bindings/RenderThread.cpp:692:3
    #21 0x7fe2fca47b7a in HandleFrameOneDoc /gecko/gfx/webrender_bindings/RenderThread.cpp:639:3
    #22 0x7fe2fca47b7a in WrNotifierEvent_HandleWakeUp /gecko/gfx/webrender_bindings/RenderThread.cpp:592:3
    #23 0x7fe2fca47b7a in mozilla::wr::RenderThread::HandleWrNotifierEvents(mozilla::wr::WrWindowId) /gecko/gfx/webrender_bindings/RenderThread.cpp:551:9
    #24 0x7fe2fca66f67 in operator()<StoreCopyPassByConstLRef<mozilla::wr::WrWindowId> &> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1164:18
    #25 0x7fe2fca66f67 in __invoke_impl<void, (lambda at /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1163:9), StoreCopyPassByConstLRef<mozilla::wr::WrWindowId> &> /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:60:14
    #26 0x7fe2fca66f67 in __invoke<(lambda at /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1163:9), StoreCopyPassByConstLRef<mozilla::wr::WrWindowId> &> /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/invoke.h:95:14
    #27 0x7fe2fca66f67 in __apply_impl<(lambda at /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1163:9), std::tuple<StoreCopyPassByConstLRef<mozilla::wr::WrWindowId> > &, 0UL> /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/tuple:1662:14
    #28 0x7fe2fca66f67 in apply<(lambda at /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1163:9), std::tuple<StoreCopyPassByConstLRef<mozilla::wr::WrWindowId> > &> /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../../include/c++/7.5.0/tuple:1671:14
    #29 0x7fe2fca66f67 in apply<mozilla::wr::RenderThread, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId)> /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1162:12
    #30 0x7fe2fca66f67 in mozilla::detail::RunnableMethodImpl<mozilla::wr::RenderThread*, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId), true, (mozilla::RunnableKind)0, mozilla::wr::WrWindowId>::Run() /builds/worker/workspace/obj-build/dist/include/nsThreadUtils.h:1213:13
    #31 0x7fe2f9df0594 in nsThread::ProcessNextEvent(bool, bool*) /gecko/xpcom/threads/nsThread.cpp:1233:16
    #32 0x7fe2f9dfa2b4 in NS_ProcessNextEvent(nsIThread*, bool) /gecko/xpcom/threads/nsThreadUtils.cpp:479:10
    #33 0x7fe2fb5e294a in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /gecko/ipc/glue/MessagePump.cpp:300:20
    #34 0x7fe2fb45ef57 in RunInternal /gecko/ipc/chromium/src/base/message_loop.cc:369:10
    #35 0x7fe2fb45ef57 in RunHandler /gecko/ipc/chromium/src/base/message_loop.cc:362:3
    #36 0x7fe2fb45ef57 in MessageLoop::Run() /gecko/ipc/chromium/src/base/message_loop.cc:344:3
    #37 0x7fe2f9de7e35 in nsThread::ThreadFunc(void*) /gecko/xpcom/threads/nsThread.cpp:391:10
    #38 0x7fe31bfeb628 in _pt_root /gecko/nsprpub/pr/src/pthreads/ptthread.c:201:5
    #39 0x7fe31cc4b608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477:8
    #40 0x7fe31c7f6132 in __clone /build/glibc-SzIz7B/glibc-2.31/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Attached file uaf-log.txt

A use-after-free with the same stack was also reported during reduction.

Verified bug as reproducible on mozilla-central 20230411035719-948cf466f3f2.
The bug appears to have been introduced in the following build range:

Start: 89d051dc90d53545631217433f34b150ef1d17bb (20230403204218)
End: cbc2d5e7ee448e665e343ccff83ca70630eea615 (20230403231556)
Pushlog: https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=89d051dc90d53545631217433f34b150ef1d17bb&tochange=cbc2d5e7ee448e665e343ccff83ca70630eea615

Keywords: regression
Whiteboard: [bugmon:bisected,confirmed]

The "use-after-free" looks like it is really a wild pointer issue. The allocation and free are for JS, while the use is for some SWGL thing.

Bug 1826204 looks like the most likely regressor in that range, however there is some other graphics-y stuff in the range.

Assignee: nobody → lsalzman

Lee, can you have a look here, please?

Flags: needinfo?(lsalzman)

Sotaro, it potentially looks like from the testcase that we're busy rendering at the time window.close is called and frees the framebuffer, so that something else can potentially reuse that memory while SWGL is busy rendering to it?

I don't see SWGL itself doing anything wrong. Is there a chance your changes in bug 1826204 could have caused something like that?

Flags: needinfo?(lsalzman) → needinfo?(sotaro.ikeda.g)
Assignee: lsalzman → nobody
Assignee: nobody → sotaro.ikeda.g
Flags: needinfo?(sotaro.ikeda.g)

Based on comment #2, this bug contains a bisection range found by bugmon. However, the Regressed by field is still not filled.

:sotaro, if possible, could you fill the Regressed by field and investigate this regression?

For more information, please visit auto_nag documentation.

Flags: needinfo?(sotaro.ikeda.g)
Flags: needinfo?(sotaro.ikeda.g)

The severity field is not set for this bug.
:gw, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(gwatson)

I could reproduce the problem with local build like the following

  • [1] Update prefs
    • set pref dom.popup_maximum = 200
    • set pref gfx.webrender.software = true
  • [2] ./mach run testcase.html

If [2] does not cause the problem. Repeat [2] until the problem happens.

When the problem happened, the following happened

From Attachment 9330537 [details], AsyncImagePipelineManager kept ShmenTextureHost alive as expected. But WebRender seemed to use the ShmenTextureHost longer.

one example sequence was like the following.

  • [epoch 3] WebRenderBridgeParent::PushExternalImageForTexture() was called in WebRenderBridgeParent::RecvSetDisplayList()
    • ShmemTexureHost is held by non-root WebRenderBridgeParent.
  • [epoch 4] The non-root WebRenderBridgeParent was destroyed by calling WebRenderBridgeParent::Destroy()
    • Its resources were cleared by WebRenderBridgeParent::ClearResources()
    • Its pipeline was removed.
    • The ShmemTexure was held until its pipeline removal
  • // WebRender processing.
  • AsyncImagePipelineManager::ProcessPipelineRemoved() confirmed a pipeline remove of the WebRenderBridgeParent.
    • The ProcessPipelineRemoved() moved the ShmemTexure to AsyncImagePipelineManager::mTexturesInUseByGPU
  • New RenderedFrameId update the mTexturesInUseByGPU in AsyncImagePipelineManager::CheckForTextureHostsNotUsedByGPU()
    • The ShmemTexure was removed from the mTexturesInUseByGPU.
    • The ShmemTexure was destroyed.
  • software WebRender uses buffer of RenderBufferTextureHost of the ShmemTexure
    • It caused the crash.

AsyncImagePipelineManager does not expect TextureHost usage after its belonging pipeline removal. But there was a case that happens.

I wonder if a problem exist in WebRender.

This might be similar to Bug 1828726.

:gw, do you have any ideas why comment 14 and comment 15 happen?

The regression range [1] has the RenderNotifier refactor patch [2]. This seems like it probably is the cause? I'd probably see if the test case reproduces with that patch reverted.

[1] https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=89d051dc90d53545631217433f34b150ef1d17bb&tochange=cbc2d5e7ee448e665e343ccff83ca70630eea615

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1826204

Flags: needinfo?(gwatson) → needinfo?(sotaro.ikeda.g)

The problem happened by calling WebRenderAPI::UpdateDebugFlags() by non-root WebRenderBridgeParent.

wr transaction of non-root WebRenderBridgeParent does not directly trigger WebRender frame build. But WebRenderAPI::UpdateDebugFlags() by non-root WebRenderBridgeParent triggers notifier.wake_up(). WebRenderAPI::UpdateDebugFlags() does not trigger frame build. Then obsoleted image was used in the STR.

The problem could be avoided by permitting WebRenderAPI::UpdateDebugFlags() only from root WebRenderBridgeParent. Actually WebRenderAPI::UpdateDebugFlags() call from non-root WebRenderBridgeParent is not necessary.

Flags: needinfo?(sotaro.ikeda.g)

Root cause was added by Bug 1600472. Then the frequency of occurrence increased by Bug 1826204.

Regressed by: 1600472, 1826204

Comment on attachment 9332307 [details]
Bug 1827340 - Update WebRender settings only from root WebRenderBridgeParent

Security Approval Request

  • How easily could an exploit be constructed based on the patch?: It is hard.
  • Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
  • Which older supported branches are affected by this flaw?: 72
  • If not all supported branches, which bug introduced the flaw?: Bug 1600472
  • Do you have backports for the affected branches?: No
  • If not, how different, hard to create, and risky will they be?: It is easy to create a patch for branches.
  • How likely is this patch to cause regressions; how much testing does it need?: It is not likely to cause regressions. With the patch, non-root WebRenderBridgeParent will only stop updating WebRender settings.
  • Is Android affected?: No
Attachment #9332307 - Flags: sec-approval?

Comment on attachment 9332307 [details]
Bug 1827340 - Update WebRender settings only from root WebRenderBridgeParent

Approved to land and uplift

Attachment #9332307 - Flags: sec-approval? → sec-approval+

Comment on attachment 9332307 [details]
Bug 1827340 - Update WebRender settings only from root WebRenderBridgeParent

Beta/Release Uplift Approval Request

  • User impact if declined: Crash might happen when a tab is opened and then quickly closed.
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: none
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): With the patch, non-root WebRenderBridgeParent will only stop updating WebRender settings.
  • String changes made/needed: none
  • Is Android affected?: No
Attachment #9332307 - Flags: approval-mozilla-beta?
Group: gfx-core-security → core-security-release
Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → 115 Branch

Verified bug as fixed on rev mozilla-central 20230512040642-55608cb73889.

Status: RESOLVED → VERIFIED

We'll need a rebased patch for ESR w/ approval request also when you get a chance.

Flags: needinfo?(sotaro.ikeda.g)

Comment on attachment 9332307 [details]
Bug 1827340 - Update WebRender settings only from root WebRenderBridgeParent

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration:
  • User impact if declined: Crash might happen when a tab is opened and then quickly closed.
  • Fix Landed on Version: 115
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): It is not likely to cause regressions. With the patch, non-root WebRenderBridgeParent will only stop updating WebRender settings.
Flags: needinfo?(sotaro.ikeda.g)
Attachment #9332307 - Flags: approval-mozilla-esr102?
Attachment #9333550 - Flags: approval-mozilla-esr102?
Attachment #9332307 - Flags: approval-mozilla-esr102?

Comment on attachment 9332307 [details]
Bug 1827340 - Update WebRender settings only from root WebRenderBridgeParent

Approved for 114.0b6.

Attachment #9332307 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

Comment on attachment 9333550 [details]
Bug 1827340 - patch for ESR

Approved for 102.12esr.

Attachment #9333550 - Flags: approval-mozilla-esr102? → approval-mozilla-esr102+
Blocks: 1828726
Flags: qe-verify-
Whiteboard: [bugmon:bisected,confirmed] → [bugmon:bisected,confirmed][adv-main114+r]
Whiteboard: [bugmon:bisected,confirmed][adv-main114+r] → [bugmon:bisected,confirmed][adv-main114+r][adv-esr102.12+r]
Duplicate of this bug: 1828726

Copying crash signatures from duplicate bugs.

Crash Signature: [@ MacIOSurface::~MacIOSurface]
Group: core-security-release
Assignee: sotaro.ikeda.g → nobody
Keywords: bugmon
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: