1423833 - Crash in shutdownhang | PR_MD_WAIT_CV | PR_Wait | mozilla::ReentrantMonitor::Wait | mozilla::layers::SynchronousTask::Wait

Status: RESOLVED FIXED

(Core :: Graphics: Layers, defect, P3)

Description

[Sylvestre Ledru [:Sylvestre]]

193 crashes yesterday on beta

This bug was filed from the Socorro interface and is
report bp-66df609b-2484-44b7-8f07-59f5a0171207.
=============================================================

Top 10 frames of crashing thread:

0 ntdll.dll NtWaitForSingleObject 
1 kernelbase.dll WaitForSingleObjectEx 
2 nss3.dll PR_MD_WAIT_CV nsprpub/pr/src/md/windows/w95cv.c:248
3 nss3.dll PR_WaitCondVar nsprpub/pr/src/threads/combined/prucv.c:172
4 nss3.dll PR_WaitCondVar nsprpub/pr/src/threads/combined/prucv.c:525
5 nss3.dll PR_Wait nsprpub/pr/src/threads/prmon.c:298
6 xul.dll mozilla::ReentrantMonitor::Wait xpcom/threads/ReentrantMonitor.h:91
7 xul.dll mozilla::layers::SynchronousTask::Wait gfx/layers/ipc/SynchronousTask.h:29
8 xul.dll mozilla::layers::ImageBridgeChild::WillShutdown gfx/layers/ipc/ImageBridgeChild.cpp:642
9 xul.dll mozilla::layers::ImageBridgeChild::ShutdownSingleton gfx/layers/ipc/ImageBridgeChild.cpp:625

=============================================================

Comment 1

[Morris Tseng [:mtseng] [:Morris] (Inactive)]

Looks like a shutdown hang in ImageBridgeChild. nical, do you have idea for this?

Comment 2

[Nicolas Silva [:nical]]

The main thread is killed because shutdown took too long. The stacks show that the Main thread is waiting for the ImageBridgeChild thread while the latter is waiting for the compositor (through a sync IPDL message). The compositor is sitting in its event loop doing nothing (it has either not picked up the message yet or has already responded by the child side hasn't received it). The IPDL thread is sitting empty handed on its event loop.

From there it's unclear whether something deadlocked (if so that's inside IPDL's handling of sync messages which would be very scary, I'm tempted to assume that's not the case), or if the shutdown is indeed just taking a long time.
Shutting gecko down is a terribly interdependent and synchronous mess and (off hands) I can't think of a way to speed it up (without leaking and) without months of work involving most of the modules that are in ShutdownXPCOM. Nothing that would be sane to uplift to beta for sure.

It's interesting that this seem to have increased dramatically in a release that would have shipped around October 12nd or so. It would be worth looking into what has landed around this train that may have slowed down ShutdownXPCOM (not necessarily only the ImageBridge part).

Comment 3

[Hani Yacoub, Desktop QA]

Closing this as resolved:worksforme since there were no crashes in the last 6 months for this signature.

Comment 4

[Hani Yacoub, Desktop QA]

I'll Reopen this bug again. There have been crashes with this signature.

Comment 6

[Brad Werth [:bradwerth]]

Attachment: Bug 1423833: Give WinWindowOcclusionTracker safe shutdown timeout semantics.

This follows similar shutdown timeout logic as used in
WinCompositorWindowThread. It waits a reasonable amount of time, but will
leak memory "safely" if timeout occurs, without crashing.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/97c2d1419843
Give WinWindowOcclusionTracker safe shutdown timeout semantics. r=rkraesig

Comment 8

[Natalia Csoregi [:nataliaCs]]

Backed out for assertion failures on WinWindowOcclusionTracker.cpp

• backout: https://hg.mozilla.org/integration/autoland/rev/14fe31be0346443253edee063dccdedaecbefbb1
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=KGANL_lRQRa03VizK0LgTQ.0&revision=97c2d1419843c7e92780ecafc010aa9272eaba3a
• failure log: https://treeherder.mozilla.org/logviewer?job_id=403314716&repo=autoland&lineNumber=1842

[task 2023-01-24T01:04:58.039Z] 01:04:58     INFO -  Assertion failure: sTracker, at /builds/worker/checkouts/gecko/widget/windows/WinWindowOcclusionTracker.cpp:374
[task 2023-01-24T01:04:58.044Z] 01:04:58     INFO -  [GPU 3640, Main Thread] WARNING: NS_ENSURE_TRUE(Preferences::InitStaticMembers()) failed: file /builds/worker/checkouts/gecko/modules/libpref/Preferences.cpp:4663
[task 2023-01-24T01:04:58.190Z] 01:04:58     INFO -  #01: soundtouch::SoundTouch::operator=[Z:\task_167452120582472\build\application\firefox\xul.dll +0x4dd9ef3]
[task 2023-01-24T01:04:58.194Z] 01:04:58     INFO -  #02: DumpCompleteHeap[Z:\task_167452120582472\build\application\firefox\xul.dll +0x1672e10]
[task 2023-01-24T01:04:58.194Z] 01:04:58     INFO -  #03: Ordinal0[Z:\task_167452120582472\build\application\firefox\xul.dll +0x441d94]
[task 2023-01-24T01:04:58.195Z] 01:04:58     INFO -  #04: workerlz4_maxCompressedSize[Z:\task_167452120582472\build\application\firefox\xul.dll +0x6f88476]
[task 2023-01-24T01:04:58.195Z] 01:04:58     INFO -  #05: workerlz4_maxCompressedSize[Z:\task_167452120582472\build\application\firefox\xul.dll +0x6f97ec2]
[task 2023-01-24T01:04:58.195Z] 01:04:58     INFO -  #06: workerlz4_maxCompressedSize[Z:\task_167452120582472\build\application\firefox\xul.dll +0x6f98698]
[task 2023-01-24T01:04:58.195Z] 01:04:58     INFO -  #07: Ordinal0[Z:\task_167452120582472\build\application\firefox\firefox.exe +0x195d]
[task 2023-01-24T01:04:58.195Z] 01:04:58     INFO -  #08: Ordinal0[Z:\task_167452120582472\build\application\firefox\firefox.exe +0x12dd]
[task 2023-01-24T01:04:58.195Z] 01:04:58     INFO -  #09: TargetNtUnmapViewOfSection[Z:\task_167452120582472\build\application\firefox\firefox.exe +0xa1678]
[task 2023-01-24T01:04:58.196Z] 01:04:58     INFO -  #10: BaseThreadInitThunk[C:\Windows\System32\KERNEL32.DLL +0x17034]
[task 2023-01-24T01:04:58.196Z] 01:04:58     INFO -  #11: RtlUserThreadStart[C:\Windows\SYSTEM32\ntdll.dll +0x52651]

Comment 9

[Natalia Csoregi [:nataliaCs]]

Also failing: PROCESS-CRASH | testing/firefox-ui/tests/functional/security/test_ssl_status_after_restart.py TestSSLStatusAfterRestart.test_ssl_status_after_restart | application crashed [@ RefPtr<mozilla::widget::DisplayStatusObserver>::operator bool() const]

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d1ac2555a5b6
Give WinWindowOcclusionTracker safe shutdown timeout semantics. r=rkraesig

Comment 11

[Sandor Molnar[:smolnar]]

Backed out for causing for causing gtest failures in WinWindowOcclusionTrackerInteractiveTest.PartialOcclusion

Backout link: https://hg.mozilla.org/integration/autoland/rev/b80b0622d26e18ecf0ea7ee79ce81a0316ab672b

Push with failures

Failure log

WARNING -  TEST-UNEXPECTED-FAIL | WinWindowOcclusionTrackerInteractiveTest.PartialOcclusion | Expected equality of these values:
[task 2023-01-24T06:55:32.475Z] 06:55:32     INFO -    nullptr
[task 2023-01-24T06:55:32.475Z] 06:55:32     INFO -      Which is: NULL
[task 2023-01-24T06:55:32.475Z] 06:55:32     INFO -    WinWindowOcclusionTracker::Get()
[task 2023-01-24T06:55:32.476Z] 06:55:32     INFO -      Which is: 0000020D560D1F00 @ /builds/worker/checkouts/gecko/widget/tests/gtest/TestWinWindowOcclusionTrackerInteractive.cpp:45
[task 2023-01-24T06:55:32.476Z] 06:55:32  WARNING -  TEST-UNEXPECTED-FAIL | WinWindowOcclusionTrackerInteractiveTest.PartialOcclusion | test completed (time: 2030ms)

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/658ab90f9614
Give WinWindowOcclusionTracker safe shutdown timeout semantics. r=rkraesig

Comment 13

[Marian-Vasile Laza]

Backed out changeset 658ab90f9614 (Bug 1423833) for causing Gtest-1proc failures.
Backout link
Push with failures <--> GTest-1proc
Failure Log
Also failure log
Also failure log
Also failure log
Also failure log
...

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/78deb7b433d8
Give WinWindowOcclusionTracker safe shutdown timeout semantics. r=rkraesig

Comment 15

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/78deb7b433d8