Closed Bug 1971452 Opened 1 year ago Closed 5 months ago

WebGPU hang after running pipeline_bind_group_compat CTS tests

Categories

(Core :: Graphics: WebGPU, defect, P3)

Product:
Core
Component:
Graphics: WebGPU
Platform:
ARM64
macOS
Type:
defect

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: aleiserson, Unassigned)

References

(
URL
)

Details

Steps to reproduce:

  1. Navigate to https://gpuweb.github.io/cts/standalone/?q=webgpu:api,validation,encoding,*
  2. Find the section for webgpu:api,validation,encoding,programmable,pipeline_bind_group_compat:* and click the play button to run it. (I am not sure how special these tests are -- I may have seen this behavior before, and if I did, it was probably different CTS tests.)
  3. Click the stop button in the upper left while the tests are running.
  4. Run any test (not sure if this is necessary)

WebGPU seems to hang. The CTS shows the blue bar at the top and does not make progress nor respond to the stop button. Trying to use WebGPU in a new tab also doesn't work. The browser also hangs on shutdown and does not exit within ~30s.

I am seeing this on Mac M1, I don't know if it is reproducible on other platforms.

Priority: P3 → P2

I can also get hangs running this under deno, where I captured the following stack trace:

  * frame #0: 0x00000001958d4bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
    frame #1: 0x0000000195760960 libdispatch.dylib`_dispatch_sema4_wait + 28
    frame #2: 0x0000000195760f10 libdispatch.dylib`_dispatch_semaphore_wait_slow + 132
    frame #3: 0x00000001a0d1d90c Metal`-[_MTLCommandBuffer initWithQueue:retainedReferences:synchronousDebugMode:] + 188
    frame #4: 0x00000001b76efc18 IOGPU`-[IOGPUMetalCommandBuffer initWithQueue:retainedReferences:synchronousDebugMode:] + 112
    frame #5: 0x000000012cb81850 AGXMetalG13X`-[AGXG13XFamilyCommandBuffer initWithQueue:retainedReferences:] + 64
    frame #6: 0x000000012cb83b3c AGXMetalG13X`-[AGXG13XFamilyCommandQueue commandBufferWithUnretainedReferences] + 44
    frame #7: 0x0000000100d5429c cts_runner`_$LT$$LP$$RP$$u20$as$u20$objc..message..MessageArguments$GT$::invoke::hb06401414d1ab05b(imp=0x0000000195525800, obj=0x0000000146854200, sel=Sel @ 0x000000016fde9dd0, (null)=<unavailable>) at mod.rs:128:17
    frame #8: 0x0000000100d40224 cts_runner`objc::message::platform::send_unverified::hde2a29210da302c2(obj=0x0000000146854200, sel=Sel @ 0x000000016fde9e48, args=<unavailable>) at mod.rs:27:9
    frame #9: 0x0000000100d67c54 cts_runner`metal::commandqueue::CommandQueueRef::new_command_buffer_with_unretained_references::hb521c17409a2c9c7 [inlined] objc::message::send_message::hba0a7264c16766e3(obj=0x0000000146854200, sel=Sel @ 0x000000016fde9f30, args=<unavailable>) at mod.rs:178:5
    frame #10: 0x0000000100d67c38 cts_runner`metal::commandqueue::CommandQueueRef::new_command_buffer_with_unretained_references::hb521c17409a2c9c7(self=0x0000000146854200) at commandqueue.rs:38:18
    frame #11: 0x0000000100967a30 cts_runner`wgpu_hal::metal::command::_$LT$impl$u20$wgpu_hal..CommandEncoder$u20$for$u20$wgpu_hal..metal..CommandEncoder$GT$::begin_encoding::_$u7b$$u7b$closure$u7d$$u7d$::h07c06e919380e22b at command.rs:200:17
    frame #12: 0x000000010091bbbc cts_runner`objc::rc::autorelease::autoreleasepool::h6662bc4fa399e254(f={closure_env#0} @ 0x000000016fdea0a8) at autorelease.rs:29:5
    frame #13: 0x0000000100925a7c cts_runner`wgpu_hal::metal::command::_$LT$impl$u20$wgpu_hal..CommandEncoder$u20$for$u20$wgpu_hal..metal..CommandEncoder$GT$::begin_encoding::h21d02938cac7778d(self=0x0000000386b2b410, label=Option<&str> @ 0x000000016fdea0d8) at command.rs:196:19
    frame #14: 0x00000001007ff5a0 cts_runner`_$LT$C$u20$as$u20$wgpu_hal..dynamic..command..DynCommandEncoder$GT$::begin_encoding::hcf2719fc0aaa6f98(self=0x0000000386b2b410, label=Option<&str> @ 0x000000016fdea130) at command.rs:219:18
    frame #15: 0x00000001005b7328 cts_runner`wgpu_core::command::CommandEncoder::open_pass::h52581d44c5737fe8(self=0x0000000386b2b630, label=Option<&str> @ 0x000000016fdea228) at mod.rs:537:18
    frame #16: 0x00000001006e8d08 cts_runner`wgpu_core::command::render::_$LT$impl$u20$wgpu_core..global..Global$GT$::render_pass_end::h99599b2e13949908(self=0x0000000147018410, pass=0x000004ac00410670) at render.rs:2068:27
    frame #17: 0x00000001004611c4 cts_runner`_$LT$deno_webgpu..render_pass..GPURenderPassEncoder$u20$as$u20$deno_webgpu..render_pass..GPURenderPassEncoder..end..Callable$GT$::call::h9941ba4e7747576d(self=0x000004ac00410640) at render_pass.rs:139:19
    frame #18: 0x0000000100460edc cts_runner`deno_webgpu::render_pass::GPURenderPassEncoder::end::end::slow_function_impl::h666fdd5de8adeec5(info=0x000000016fdf2308) at render_pass.rs:33:1
    frame #19: 0x0000000100460f88 cts_runner`deno_webgpu::render_pass::GPURenderPassEncoder::end::end::v8_fn_ptr::h6a3897b49c99ef50(info=0x000000016fdf2308) at render_pass.rs:33:1

My theory here is that we are accumulating too many command buffers that are valid but unsubmitted and won't be cleaned up until garbage collection. When we exhaust the available command buffers, Metal blocks, on the assumption that command buffers will be freed up shortly when GPU execution completes.

makecommandqueue() says you can have up to 64 "uncompleted" command buffers, and makecommandbuffer() says:

This method blocks the calling CPU thread when the queue doesn’t have any free command buffers, and returns after the GPU finishes executing one.

From counting the command buffers at runtime, the limit we are hitting appears to be 2048.

groves pointed out on Matrix that we set the limit to 2048.

wgpu PR: https://github.com/gfx-rs/wgpu/pull/7858

OS: Unspecified → macOS
Hardware: Unspecified → ARM64
Depends on: 1974980

The linked change to increase the limit to 4,096 has not resolved the problem with this test suite in Firefox.

I did capture the stack trace from the hung process on shutdown, which confirms the hang in Firefox is due to Metal command buffer exhaustion. Since this is unlikely to affect real applications, I am going to lower the priority.

  Thread 0x250f92f    Thread name "CanvasRenderer"    18 samples (1-18)    priority 31 (base 31)
  <process frontmost, thread QoS default (requested default), process unclamped, process received importance donation from WindowServer [587], process received importance donation from CursorUIViewService [1301], IO tier 0>
  18  thread_start + 8 (libsystem_pthread.dylib + 7040) [0x195911b80] 1-18
    18  _pthread_start + 136 (libsystem_pthread.dylib + 27660) [0x195916c0c] 1-18
      18  _pt_root + 260 (libnss3.dylib + 814656) [0x103f36e40] 1-18
        18  nsThread::ThreadFunc(void*) + 272 (XUL + 1919728) [0x11ea10af0] 1-18
          18  MessageLoop::Run() + 80 (XUL + 4618944) [0x11eca3ac0] 1-18
            18  mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) + 412 (XUL + 4881944) [0x11ece3e18] 1-18
              18  NS_ProcessNextEvent(nsIThread*, bool) + 2544 (XUL + 1952268) [0x11ea18a0c] 1-18
                18  mozilla::ipc::MessageChannel::MessageTask::Run() + 168 (XUL + 4866488) [0x11ece01b8] 1-18
                  18  mozilla::ipc::MessageChannel::RunMessage(mozilla::ipc::ActorLifecycleProxy*, mozilla::ipc::MessageChannel::MessageTask&) + 796 (XUL + 4860920) [0x11ecdebf8] 1-18
                    18  mozilla::gfx::PCanvasManagerParent::OnMessageReceived(IPC::Message const&) + 104 (XUL + 8486348) [0x11f053dcc] 1-18
                      18  mozilla::webgpu::PWebGPUParent::OnMessageReceived(IPC::Message const&) + 816 (XUL + 64160540) [0x12256c31c] 1-18
                        18  mozilla::webgpu::WebGPUParent::RecvMessages(unsigned int, mozilla::ipc::ByteBuf&&, nsTArray<mozilla::ipc::ByteBuf>&&, nsTArray<mozilla::ipc::shared_memory::Handle<(mozilla::ipc::shared_memory::Type)0>>&&) + 328 (XUL + 64157412) [0x12256b6e4] 1-18
                          18  wgpu_server_messages + 3276 (XUL + 94934024) [0x1242c5408] 1-18
                            18  wgpu_bindings::command::replay_render_pass::hf79f94f54c9f9344 + 316 (XUL + 94907440) [0x1242bec30] 1-18
                              18  wgpu_core::command::render::_$LT$impl$u20$wgpu_core..global..Global$GT$::render_pass_end::h82204381eddaa77c + 636 (XUL + 95669448) [0x124378cc8] 1-18
                                18  wgpu_core::command::render::_$LT$impl$u20$wgpu_core..global..Global$GT$::render_pass_end::_$u7b$$u7b$closure$u7d$$u7d$::hb24ee8f7dd0d7386 + 364 (XUL + 95670052) [0x124378f24] 1-18
                                  18  wgpu_core::command::CommandEncoder::open_pass::hda2edb13a43fcd5c + 76 (XUL + 95734396) [0x124388a7c] 1-18
                                    18  wgpu_hal::metal::command::_$LT$impl$u20$wgpu_hal..CommandEncoder$u20$for$u20$wgpu_hal..metal..CommandEncoder$GT$::begin_encoding::h01f957db143afdcc + 92 (XUL + 96185516) [0x1243f6cac] 1-18
                                      18  -[AGXG13XFamilyCommandQueue commandBuffer] + 44 (AGXMetalG13X + 1850244) [0x12fd87b84] 1-18
                                        18  -[AGXG13XFamilyCommandBuffer initWithQueue:retainedReferences:] + 64 (AGXMetalG13X + 1841232) [0x12fd85850] 1-18
                                          18  -[IOGPUMetalCommandBuffer initWithQueue:retainedReferences:synchronousDebugMode:] + 112 (IOGPU + 11288) [0x1b76efc18] 1-18
                                            18  -[_MTLCommandBuffer initWithQueue:retainedReferences:synchronousDebugMode:] + 188 (Metal + 92428) [0x1a0d1d90c] 1-18
                                              18  _dispatch_semaphore_wait_slow + 132 (libdispatch.dylib + 16144) [0x195760f10] 1-18
                                                18  semaphore_wait_trap + 8 (libsystem_kernel.dylib + 2992) [0x1958d4bb0] 1-18
                                                 *1   ??? (kernel.release.t6000 + 694316) [0xfffffe000884d82c] 1
  <process non-frontmost>
                                                 *17  ??? (kernel.release.t6000 + 694316) [0xfffffe000884d82c] 2-18
No longer depends on: 1974980
Priority: P2 → P3

This was most likely fixed by https://phabricator.services.mozilla.com/D284325. I can't reproduce with either of the CTS suites mentioned above.

Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.