crash in shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup

NEW
Assigned to

Status

()

Core
DOM: Workers
P3
critical
2 years ago
2 months ago

People

(Reporter: marco, Assigned: baku)

Tracking

(4 keywords)

46 Branch
All
Windows
crash, hang, steps-wanted, topcrash-win
Points:
---

Firefox Tracking Flags

(firefox46 wontfix, firefox47 wontfix, firefox48+ wontfix, firefox49 wontfix, firefox-esr45 wontfix, firefox50 wontfix, firefox51 affected, firefox52 affected, firefox53 affected, firefox54 affected, firefox55 affected)

Details

(Whiteboard: btpp-active, crash signature)

(Reporter)

Description

2 years ago
This bug was filed from the Socorro interface and is 
report bp-3d70f587-c687-443b-bbee-001812160426.
=============================================================

Filing this bug because there is no associated report to that signature yet.

This shutdown hang is #28 top crasher (with 2546 crashes) on Firefox 45 and #28 top crasher (with 715 crashes) on Firefox 46.

There are a few reports for Firefox 47 (18), no reports for Firefox 48.
(Reporter)

Updated

2 years ago
Keywords: hang, topcrash-win
(Reporter)

Updated

2 years ago
status-firefox48: --- → ?
From one of the comments, if you go to tut.by then exit you get a hang. I was able to reproduce a shutdown hang on Nightly.
If I sampled the right process, this was happening in nsXMLHttpRequest::Send. My crash had a different signature, where the hang was in compositor bridge shutdown or something, but I think that just spins the event loop in the child until that clears out.
baku, can you please take a look?
Flags: needinfo?(amarchesini)
Whiteboard: btpp-followup-2016-05-06
(Assignee)

Comment 4

2 years ago
I see often mozilla::dom::PromiseWorkerProxy::CleanUp(JSContext*) in the crash reports:

https://hg.mozilla.org/releases/mozilla-release/annotate/e35da3da61cb/dom/promise/Promise.cpp#l2592

and many of them are related with Push. I think there is a kind of deadlock with the Promise Proxy.
I'm debugging it.
Flags: needinfo?(amarchesini)
Assignee: nobody → amarchesini
Whiteboard: btpp-followup-2016-05-06 → btpp-active

Comment 5

2 years ago
[Tracking Requested - why for this release]:
newer data suggests that it is also present on 48. it's probably too late for 47, but i'm requesting tracking for the 48 cycle, since a fix would be nice (signature is #25 on release currently).
status-firefox48: ? → affected
tracking-firefox48: --- → ?
Track this as this signature is #25.
tracking-firefox48: ? → +
Can we please get an STR?
Keywords: steps-wanted
(Assignee)

Comment 8

2 years ago
(In reply to Andrew McCreight [:mccr8] from comment #1)
> From one of the comments, if you go to tut.by then exit you get a hang. I
> was able to reproduce a shutdown hang on Nightly.

I'm not able to reproduce it. Nightly shuts down correctly.
Crash volume for signature 'shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup':
 - esr (version 45): 558 crashes from 2016-04-07.

Crash volume on the last weeks:
         Week N-1   Week N-2   Week N-3   Week N-4   Week N-5   Week N-6   Week N-7
 - esr         58         58         44         78         65         61         35

Affected platform: Windows
status-firefox-esr45: --- → affected
Low volume for esr, doesn't match the criteria, wontfix.
status-firefox-esr45: affected → wontfix
I don't see any crashes reported > 48. Could this have been fixed in 48?
Flags: needinfo?(sledru)
It seems, the question is "which bug fixed it" ? :)
Flags: needinfo?(sledru)
(In reply to Sylvestre Ledru [:sylvestre] from comment #12)
> It seems, the question is "which bug fixed it" ? :)

Any ideas, baku?
Flags: needinfo?(amarchesini)
We will ship 48 with this bug.
status-firefox46: affected → wontfix
status-firefox47: affected → wontfix
status-firefox48: affected → wontfix
status-firefox49: --- → fixed
(Assignee)

Comment 15

2 years ago
Could be bug 1288033.
Flags: needinfo?(amarchesini)
(Reporter)

Comment 16

2 years ago
There are several signatures that contain mozilla::dom::workers::RuntimeService::Cleanup.

Looks like the crash might affect 49.0b as well:
https://crash-stats.mozilla.com/search/?product=Firefox&proto_signature=~mozilla%3A%3Adom%3A%3Aworkers%3A%3ARuntimeService%3A%3ACleanup&version=49.0b&_sort=-date&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

For shutdownhang | mozilla::CondVar::Wait | nsEventQueue::GetEvent, mozilla::dom::workers::RuntimeService::Cleanup is on the stack, but I'm not sure it's the same crash.
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent …
(Reporter)

Comment 17

2 years ago
An example crash report on Beta 49: https://crash-stats.mozilla.com/report/index/93966b6e-d9b4-480c-a058-1f1e62160828#allthreads. It does look like the same crash to me.
status-firefox49: fixed → affected
status-firefox50: --- → affected
status-firefox51: --- → affected
(Reporter)

Updated

2 years ago
Depends on: 1272911

Updated

2 years ago
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent … → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent …

Comment 18

2 years ago
I've got a report from a Twitter user who is seeing this crash on close with 49.02.  Here are his IDs 
bp-1b47df2c-a300-4133-b6c2-a13752161103
bp-530e5f0c-cc06-403d-af69-c842e2161102
bp-7b9d1f1a-394d-4ec9-9073-075dc2161102

Comment 19

a year ago
these crashes seem to have gone away after nightly 52.0a1 build 20161029062601
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent … → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent …
status-firefox49: affected → wontfix
status-firefox50: affected → wontfix
status-firefox52: --- → unaffected
status-firefox53: --- → unaffected
(Reporter)

Updated

a year ago
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent … → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent …

Comment 20

a year ago
I to have a same crash in FF ESR 45.7

Crash ID :c20817fb-ae12-4acc-beca-bd9ab2170307

Updated

a year ago
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent … → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent …

Updated

a year ago
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent … → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent …

Comment 21

a year ago
crash happens while Win 8.1 is going to "stand by" after a period of beeing screensaver active. There was no direct action from my side to force a crash.

Thunderbird 52.0 Crash Report [@ shutdownhang | _PR_MD_WAIT_CV | _PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent | nsThread::nsChainedEventQueue::GetEvent | nsThread::GetEvent | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService:... ]

This bug was filed from the Socorro interface and is 
report bp-58ad40bb-c89c-4c31-ace8-2d7552170412.
=============================================================

0 	ntdll.dll 	NtWaitForSingleObject 	
1 	kernelbase.dll 	WaitForSingleObjectEx 	
2 	kernelbase.dll 	WaitForSingleObject 	
3 	nss3.dll 	_PR_MD_WAIT_CV 	nsprpub/pr/src/md/windows/w95cv.c:248
4 	nss3.dll 	_PR_WaitCondVar 	nsprpub/pr/src/threads/combined/prucv.c:172
5 	nss3.dll 	PR_WaitCondVar 	nsprpub/pr/src/threads/combined/prucv.c:525
6 	xul.dll 	mozilla::CondVar::Wait(unsigned int) 	C:/builds/moz2_slave/tb-rel-c-esr52-w32_bld-0000000/build/objdir-tb/dist/include/mozilla/CondVar.h:79
7 	xul.dll 	nsEventQueue::GetEvent(bool, nsIRunnable**, mozilla::BaseAutoLock<mozilla::Mutex>&) 	xpcom/threads/nsEventQueue.cpp:57
8 	xul.dll 	nsThread::nsChainedEventQueue::GetEvent(bool, nsIRunnable**, mozilla::BaseAutoLock<mozilla::Mutex>&) 	xpcom/threads/nsThread.cpp:788
9 	xul.dll 	nsThread::GetEvent(bool, nsIRunnable**, mozilla::BaseAutoLock<mozilla::Mutex>&) 	xpcom/threads/nsThread.cpp:1147
10 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1206
11 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/glue/nsThreadUtils.cpp:361
12 	xul.dll 	mozilla::dom::workers::RuntimeService::Cleanup() 	dom/workers/RuntimeService.cpp:2198
13 	xul.dll 	mozilla::dom::workers::RuntimeService::Observe(nsISupports*, char const*, char16_t const*) 	dom/workers/RuntimeService.cpp:2717
14 	xul.dll 	nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) 	xpcom/ds/nsObserverList.cpp:112
15 	xul.dll 	nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) 	xpcom/ds/nsObserverService.cpp:281
16 	xul.dll 	mozilla::ShutdownXPCOM(nsIServiceManager*) 	xpcom/build/XPCOMInit.cpp:926
17 	xul.dll 	ScopedXPCOMStartup::~ScopedXPCOMStartup() 	toolkit/xre/nsAppRunner.cpp:1401
18 	xul.dll 	mozilla::DefaultDelete<ScopedXPCOMStartup>::operator()(ScopedXPCOMStartup*) 	C:/builds/moz2_slave/tb-rel-c-esr52-w32_bld-0000000/build/objdir-tb/dist/include/mozilla/UniquePtr.h:528
19 	xul.dll 	XREMain::XRE_main(int, char** const, nsXREAppData const*) 	toolkit/xre/nsAppRunner.cpp:4649
20 	xul.dll 	XRE_main 	toolkit/xre/nsAppRunner.cpp:4712
21 	thunderbird.exe 	do_main 	C:/builds/moz2_slave/tb-rel-c-esr52-w32_bld-0000000/build/mail/app/nsMailApp.cpp:245
22 	thunderbird.exe 	NS_internal_main(int, char**, char**) 	C:/builds/moz2_slave/tb-rel-c-esr52-w32_bld-0000000/build/mail/app/nsMailApp.cpp:378
23 	thunderbird.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:115
24 	thunderbird.exe 	__scrt_common_main_seh 	f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:253
25 	kernel32.dll 	BaseThreadInitThunk 	
26 	ntdll.dll 	__RtlUserThreadStart 	
27 	ntdll.dll 	_RtlUserThreadStart
This may be responsible for the current crashrate spike (4x normal) we're seeing on release 53.0.3, as reported by telemetry on https://telemetry.mozilla.org/crashes/

The hangs seem to come universally from Windows 7 users and the stack is exactly as described in Comment 21.
By :marco's request, here's the ipython code for symbolicating the main thread of a crash ping

import requests

def symbolicate(s):
    data = json.dumps({
            'stacks': [[[f['module_index'], int(f['ip'], 16) - int(s['modules'][f['module_index']]['base_addr'], 16)] for f in s['threads'][0]['frames']]],
            'memoryMap': [[m['debug_file'].translate(dict((ord(char), None) for char in ' ()')), m['debug_id']] for m in s['modules']], 'version': 4})
    result = requests.post('http://symbolapi.mozilla.org/', data=data)
    result_json = result.json()
    return result_json['symbolicatedStacks']

(a cool trick is to replace [0] with [s['crash_info']['crashing_thread']] to have it symbolicate the crashing thread instead of the main thread)
(Oh, and the parameter 's' is the ping's payload.stackTraces structure.)

Updated

a year ago
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent &hellip; → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent &hellip;
status-firefox52: unaffected → affected
status-firefox53: unaffected → affected
status-firefox54: --- → affected
status-firefox55: --- → affected
(Reporter)

Updated

a year ago
See Also: → bug 1368983

Updated

11 months ago
Severity: normal → critical
(Reporter)

Updated

8 months ago
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent &hellip; → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | nsThread::ProcessNextEvent | NS_ProcessNextEvent | mozilla::dom::workers::RuntimeService::Cleanup] [@ shutdownhang | nsThread::ProcessNextEvent | NS_ProcessNextEvent &hellip;

Updated

8 months ago
See Also: → bug 1405290

Updated

4 months ago
Priority: -- → P3

Updated

2 months ago
See Also: → bug 1437575
You need to log in before you can comment on or make changes to this bug.