1745308 - Intermittent [tier 2] Main app process exited normally | application crashed [@ RaiseException + 0x62] | Shmem exhaustion after change to clientWaitSync warnings

Status: RESOLVED FIXED

(Core :: Graphics: CanvasWebGL, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: imoraru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=360700040&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/J_rMmx5ITgKLSZ8GR-vv3A/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-12-09T23:22:27.472Z] 23:22:27     INFO - TEST-START | dom/canvas/test/webgl-conf/generated/test_2_conformance2__sync__sync-webgl-specific.html
[task 2021-12-09T23:22:27.585Z] 23:22:27     INFO - GECKO(6828) | [Child 10180, Main Thread] WARNING: Not resolving response 'PDocumentChannel::Reply_RedirectToRealChannel': actor is dead: file /builds/worker/checkouts/gecko/ipc/glue/ProtocolUtils.cpp:898
[task 2021-12-09T23:22:27.810Z] 23:22:27     INFO - GECKO(6828) | [GPU 6072, Compositor] WARNING: Failed to bind API to GL!: file /builds/worker/checkouts/gecko/gfx/gl/GLContextProviderEGL.cpp:622
[task 2021-12-09T23:22:27.811Z] 23:22:27     INFO - GECKO(6828) | [GPU 6072, Compositor] WARNING: Failed to create GLContext from PBuffer: file /builds/worker/checkouts/gecko/gfx/gl/GLContextProviderEGL.cpp:1174
[task 2021-12-09T23:22:27.811Z] 23:22:27     INFO - GECKO(6828) | [GPU 6072, Compositor] WARNING: Failed to create EGLContext with khr_rbab_attribs: file /builds/worker/checkouts/gecko/gfx/gl/GLContextProviderEGL.cpp:720
[task 2021-12-09T23:22:27.812Z] 23:22:27     INFO - GECKO(6828) | Initializing context 0EB1B000 surface 01B0D3D0 on display 037B4B00
[task 2021-12-09T23:22:48.304Z] 23:22:48     INFO - GECKO(6828) | [Parent 7692, Jump List] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520012 (NS_ERROR_FILE_NOT_FOUND): file /builds/worker/checkouts/gecko/widget/windows/WinUtils.cpp:1429
[task 2021-12-09T23:23:27.302Z] 23:23:27     INFO - GECKO(6828) | [GPU 6072, BackgroundThreadPool #1] WARNING: Failed to lock the texture: file /builds/worker/checkouts/gecko/gfx/layers/d3d11/TextureD3D11.cpp:249
[task 2021-12-09T23:23:27.314Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: clientWaitSync: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.315Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: clientWaitSync: `timeout` must not exceed MAX_CLIENT_WAIT_TIMEOUT_WEBGL nanoseconds.
<...>
[task 2021-12-09T23:24:07.802Z] 23:24:07     INFO - mozcrash Saved app info as Z:\task_163909120856358\build\blobber_upload_dir\6de09154-06b3-4536-a6d5-75897630bb85.extra
[task 2021-12-09T23:24:07.941Z] 23:24:07     INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ RaiseException + 0x62]
[task 2021-12-09T23:24:07.941Z] 23:24:07     INFO - Crash dump filename: C:\Users\task_163909120856358\AppData\Local\Temp\tmp0mmy4os4.mozrunner\minidumps\6de09154-06b3-4536-a6d5-75897630bb85.dmp
[task 2021-12-09T23:24:07.941Z] 23:24:07     INFO - Operating system: Windows NT
[task 2021-12-09T23:24:07.941Z] 23:24:07     INFO -                   10.0.19041 
[task 2021-12-09T23:24:07.941Z] 23:24:07     INFO - CPU: x86
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO -      GenuineIntel family 6 model 63 stepping 2
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO -      6 CPUs
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO - 
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO - Crash reason:  unknown 0x3228369022
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO - Crash address: 0x7755b502
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO - Process uptime: 192 seconds
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO - 
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO - Thread 0 GeckoMain (crashed)
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO -  0  KERNELBASE.dll!RaiseException + 0x62
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO -     eip = 0x7755b502   esp = 0x00d78bf0   ebp = 0x00d78c48   ebx = 0x64f70000
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO -     esi = 0x64f70000   edi = 0x00000004   eax = 0x00d78bf0   ecx = 0x00000001
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO -     edx = 0x00000000   efl = 0x00000216
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO -     Found by: given as instruction pointer in context
[task 2021-12-09T23:24:07.942Z] 23:24:07     INFO -  1  mozglue.dll!__delayLoadHelper2 + 0x291
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     eip = 0x650b04e2   esp = 0x00d78c50   ebp = 0x64f70000   ebx = 0x64f70000
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     esi = 0x64f70000   edi = 0x00000004
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     Found by: call frame info
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -  2  mozglue.dll!<unknown in mozglue.dll> + 0x35e12
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     eip = 0x650e7114   esp = 0x00d78c70
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     Found by: stack scanning
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -  3  mozglue.dll!<unknown in mozglue.dll> + 0x3dee2
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     eip = 0x650ef1e4   esp = 0x00d78c74
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     Found by: stack scanning
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -  4  mozglue.dll!<unknown in mozglue.dll> + 0x36368
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     eip = 0x650e766a   esp = 0x00d78c78
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     Found by: stack scanning
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -  5  mozglue.dll!<unknown in mozglue.dll> + 0x3601a
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     eip = 0x650e731c   esp = 0x00d78c80
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     Found by: stack scanning
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -  6  mozglue.dll!mozalloc_abort(char const* const) [mozalloc_abort.cpp:7366288ec078ae49ddb523130233bd035a91e07d : 26 + 0x18]
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     eip = 0x64f925db   esp = 0x00d78ca0   ebp = 0x00d78e04
[task 2021-12-09T23:24:07.943Z] 23:24:07     INFO -     Found by: stack scanning
[task 2021-12-09T23:24:07.944Z] 23:24:07     INFO - 

Comment 1

[Iulian Moraru]

Hi Jared! This is related to the perma tier 2 Bug 1745304. Some retriggers failed with this crash. The crash appears after TEST-START | dom/canvas/test/webgl-conf/generated/test_2_conformance2__sync__sync-webgl-specific.html same as the test that fails for the perma bug.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1733106

Comment 3

[Jared Hirsch [:jhirsch] (he/him) (Needinfo please)]

Hmm. I'm not sure how best to proceed here. Kelsey, any ideas?

Comment 4

[Intermittent Failures Robot]

12 failures in 3319 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 1

Platform and build breakdown:

• windows10-32-2004-mingwclang-qr: 12
  • debug: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745308&startday=2021-12-06&endday=2021-12-12&tree=all

Comment 5

[Kelsey Gilbert [:jgilbert]]

[task 2021-12-09T23:23:27.341Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: getSyncParameter: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.342Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: getSyncParameter: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.343Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: getSyncParameter: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.344Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: getSyncParameter: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.344Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: getSyncParameter: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.346Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: getSyncParameter: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.346Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: getSyncParameter: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.346Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: WebGL warning: getSyncParameter: ClientWaitSync must return TIMEOUT_EXPIRED until control has returned to the user agent's main loop.
[task 2021-12-09T23:23:27.347Z] 23:23:27     INFO - GECKO(6828) | JavaScript warning: , line 0: After reporting 32, no further warnings will be reported for this WebGL context.
[task 2021-12-09T23:23:57.665Z] 23:23:57     INFO - GECKO(6828) | [Child 10180, Main Thread] WARNING: Failed to alloc shmem for AllocPendingCmdBytes.: file /builds/worker/checkouts/gecko/dom/canvas/WebGLChild.cpp:37
[task 2021-12-09T23:23:57.665Z] 23:23:57     INFO - GECKO(6828) | JavaScript warning: http://mochi.test:8888/tests/dom/canvas/test/webgl-conf/checkout/conformance2/sync/sync-webgl-specific.html?webglVersion=2, line 122: Failed to allocate internal command buffer.
[task 2021-12-09T23:23:57.667Z] 23:23:57     INFO - GECKO(6828) | JavaScript warning: http://mochi.test:8888/tests/dom/canvas/test/webgl-conf/checkout/conformance2/sync/sync-webgl-specific.html?webglVersion=2, line 122: WebGL context was lost.
[task 2021-12-09T23:23:57.668Z] 23:23:57     INFO - GECKO(6828) | [Child 10180, Main Thread] WARNING: Attempt to __delete__ missing or closed actor: file /builds/worker/workspace/obj-build/ipc/ipdl/PWebGLChild.cpp:110
[task 2021-12-09T23:23:57.668Z] 23:23:57     INFO - GECKO(6828) | Destroying context 0EB1B000 surface 01B0D3D0 on display 037B4B00
[task 2021-12-09T23:23:57.680Z] 23:23:57     INFO - GECKO(6828) | out of memory: 0x0000000000009BA6 bytes requested
[task 2021-12-09T23:23:57.682Z] 23:23:57     INFO - GECKO(6828) | Hit MOZ_CRASH(out of memory: 0x0000000000009BA6 bytes requested) at /builds/worker/checkouts/gecko/memory/mozalloc/mozalloc_abort.cpp:35

We're spinning the loop too hard here, and running out of shmems! (or memory?) Let me look at that patch and test again...

Comment 6

[Kelsey Gilbert [:jgilbert]]

Yeah, I think we're literally sucking up all available memory to send "please don't" warnings. :)
We should throttle this somehow. Maybe only send it once per sync object?

The alternative is disabling that test for now, but a basic throttle is probably easy.

Comment 7

[Kelsey Gilbert [:jgilbert]]

Attachment: Bug 1745308 - Only warn once if ClientWaitSync cannot return availability.

This avoids spamming warnings across IPC that sometimes causes shmem
exhaustion.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0f43d0f3d542
Only warn once if ClientWaitSync cannot return availability. r=gfx-reviewers,nical

Comment 9

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/0f43d0f3d542

Comment 10

[Intermittent Failures Robot]

2 failures in 3263 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• windows10-32-2004-mingwclang-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1745308&startday=2021-12-13&endday=2021-12-19&tree=all