1375726 - Crash in shutdownhang | _PR_MD_WAIT_CV | _PR_WaitCondVar | mozilla::CondVar::Wait | nsEventQueue::GetEvent | nsThread::nsChainedEventQueue::GetEvent | nsThread::ProcessNextEvent | NS_ProcessNextEvent | nsThread::Shutdown | mozilla::net::nsSocketTranspo...

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[Gerry Chang [:gchang]]

This bug was filed from the Socorro interface and is 
report bp-eea45607-9f68-4ffb-b62e-ad4500170622.
=============================================================
Frame 	Module 	Signature 	Source
0 	ntdll.dll 	NtWaitForSingleObject 	
1 	kernelbase.dll 	WaitForSingleObjectEx 	
2 	kernel32.dll 	WaitForSingleObjectExImplementation 	
3 	kernel32.dll 	WaitForSingleObject 	
4 	nss3.dll 	_PR_MD_WAIT_CV 	nsprpub/pr/src/md/windows/w95cv.c:248
5 	nss3.dll 	_PR_WaitCondVar 	nsprpub/pr/src/threads/combined/prucv.c:172
6 	nss3.dll 	PR_WaitCondVar 	nsprpub/pr/src/threads/combined/prucv.c:525
7 	xul.dll 	mozilla::CondVar::Wait(unsigned int) 	obj-firefox/dist/include/mozilla/CondVar.h:79
8 	xul.dll 	nsEventQueue::GetEvent(bool, nsIRunnable**, mozilla::BaseAutoLock<mozilla::Mutex>&) 	xpcom/threads/nsEventQueue.cpp:76
9 	xul.dll 	nsThread::nsChainedEventQueue::GetEvent(bool, nsIRunnable**, mozilla::BaseAutoLock<mozilla::Mutex>&) 	xpcom/threads/nsThread.cpp:820
10 	xul.dll 	nsThread::ProcessNextEvent(bool, bool*) 	xpcom/threads/nsThread.cpp:1237
11 	xul.dll 	NS_ProcessNextEvent(nsIThread*, bool) 	xpcom/threads/nsThreadUtils.cpp:389
12 	xul.dll 	nsThread::Shutdown() 	xpcom/threads/nsThread.cpp:1015
13 	xul.dll 	mozilla::net::nsSocketTransportService::ShutdownThread() 	netwerk/base/nsSocketTransportService2.cpp:612
14 	xul.dll 	mozilla::net::nsSocketTransportService::Shutdown(bool) 	netwerk/base/nsSocketTransportService2.cpp:595
15 	xul.dll 	mozilla::net::nsIOService::SetOffline(bool) 	netwerk/base/nsIOService.cpp:1117
16 	xul.dll 	mozilla::net::nsIOService::Observe(nsISupports*, char const*, char16_t const*) 	netwerk/base/nsIOService.cpp:1430
17 	xul.dll 	nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) 	xpcom/ds/nsObserverList.cpp:112
18 	xul.dll 	nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) 	xpcom/ds/nsObserverService.cpp:281
19 	xul.dll 	nsXREDirProvider::DoShutdown() 	toolkit/xre/nsXREDirProvider.cpp:1237
20 	xul.dll 	ScopedXPCOMStartup::~ScopedXPCOMStartup() 	toolkit/xre/nsAppRunner.cpp:1417
21 	xul.dll 	mozilla::UniquePtr<ScopedXPCOMStartup, mozilla::DefaultDelete<ScopedXPCOMStartup> >::reset(ScopedXPCOMStartup*) 	obj-firefox/dist/include/mozilla/UniquePtr.h:345
22 	xul.dll 	mozilla::UniquePtr<ScopedXPCOMStartup, mozilla::DefaultDelete<ScopedXPCOMStartup> >::operator=(std::nullptr_t) 	obj-firefox/dist/include/mozilla/UniquePtr.h:313
23 	xul.dll 	XREMain::XRE_main(int, char** const, mozilla::BootstrapConfig const&) 	toolkit/xre/nsAppRunner.cpp:4705
24 	xul.dll 	XRE_main(int, char** const, mozilla::BootstrapConfig const&) 	toolkit/xre/nsAppRunner.cpp:4768
25 	xul.dll 	mozilla::BootstrapImpl::XRE_main(int, char** const, mozilla::BootstrapConfig const&) 	toolkit/xre/Bootstrap.cpp:45
26 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:115
27 	firefox.exe 	__scrt_common_main_seh 	f:/dd/vctools/crt/vcstartup/src/startup/exe_common.inl:253
28 	kernel32.dll 	BaseThreadInitThunk 	
29 	ntdll.dll 	__RtlUserThreadStart 	
30 	ntdll.dll 	_RtlUserThreadStart

This is #13 topcrash of FF54 and it has a spike in the last 3 days.

Comment 1

[Gerry Chang [:gchang]]

Hi Nathan,
Can you help find someone to look at this?

Comment 2

[Nathan Froyd [:froydnj]]

It looks like the socket transport service is waiting on threads to shutdown; the crash report in comment 0 has:

Thread 7
Frame 	Module 	Signature 	Source
0 	nss3.dll 	FreeArenaList 	nsprpub/lib/ds/plarena.c:192
1 	nss3.dll 	nssArena_Destroy 	security/nss/lib/base/arena.c:477
2 	nss3.dll 	nssTrustDomain_RemoveCertFromCacheLOCKED 	security/nss/lib/pki/tdcache.c:353
3 	nss3.dll 	nssCertificate_Destroy 	security/nss/lib/pki/certificate.c:114
4 	nss3.dll 	CERT_DestroyCertificate 	security/nss/lib/certdb/stanpcertdb.c:764
5 	nss3.dll 	ssl3_CleanupPeerCerts 	security/nss/lib/ssl/ssl3con.c:10601
6 	nss3.dll 	ssl3_DestroySSL3Info 	security/nss/lib/ssl/ssl3con.c:13279
7 	nss3.dll 	ssl_DestroySocketContents 	security/nss/lib/ssl/sslsock.c:408
8 	nss3.dll 	ssl_FreeSocket 	security/nss/lib/ssl/sslsock.c:447
9 	nss3.dll 	ssl_DefClose 	security/nss/lib/ssl/ssldef.c:221
10 	nss3.dll 	ssl_SecureClose 	security/nss/lib/ssl/sslsecur.c:788
11 	nss3.dll 	ssl_Close 	security/nss/lib/ssl/sslsock.c:2690
12 	xul.dll 	nsNSSSocketInfo::CloseSocketAndDestroy(nsNSSShutDownPreventionLock const&) 	security/manager/ssl/nsNSSIOLayer.cpp:936
13 	xul.dll 	nsSSLIOLayerClose 	security/manager/ssl/nsNSSIOLayer.cpp:915
14 	nss3.dll 	PR_Close 	nsprpub/pr/src/io/priometh.c:104
15 	xul.dll 	mozilla::net::nsSocketTransport::CloseSocket(PRFileDesc*, bool) 	netwerk/base/nsSocketTransport2.cpp:3231
16 	xul.dll 	mozilla::net::nsSocketTransport::ReleaseFD_Locked(PRFileDesc*) 	netwerk/base/nsSocketTransport2.cpp:1851
17 	xul.dll 	mozilla::net::nsSocketTransport::OnSocketDetached(PRFileDesc*) 	netwerk/base/nsSocketTransport2.cpp:2161
18 	xul.dll 	mozilla::net::nsSocketTransportService::DetachSocket(mozilla::net::nsSocketTransportService::SocketContext*, mozilla::net::nsSocketTransportService::SocketContext*) 	netwerk/base/nsSocketTransportService2.cpp:245
19 	xul.dll 	mozilla::net::nsSocketTransportService::DoPollIteration(mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>*) 	netwerk/base/nsSocketTransportService2.cpp:1068

while another report, https://crash-stats.mozilla.com/report/index/fcb65b9b-d65d-4bd4-83e2-bc46c0170623 has several threads that look like:

Thread 56
Frame 	Module 	Signature 	Source
0 	ntdll.dll 	NtWaitForSingleObject 	
1 	KERNELBASE.dll 	WaitForSingleObjectEx 	
2 	KERNELBASE.dll 	WaitForSingleObject 	
3 	nss3.dll 	_PR_MD_WAIT_CV 	nsprpub/pr/src/md/windows/w95cv.c:248
4 	nss3.dll 	_PR_WaitCondVar 	nsprpub/pr/src/threads/combined/prucv.c:172
5 	nss3.dll 	PR_WaitCondVar 	nsprpub/pr/src/threads/combined/prucv.c:525
6 	xul.dll 	mozilla::CondVar::Wait(unsigned int) 	obj-firefox/dist/include/mozilla/CondVar.h:79
7 	xul.dll 	nsNSSHttpRequestSession::internal_send_receive_attempt(bool&, PRPollDesc**, unsigned short*, char const**, char const**, char const**, unsigned int*) 	security/manager/ssl/nsNSSCallbacks.cpp:454
8 	xul.dll 	nsNSSHttpRequestSession::trySendAndReceiveFcn(PRPollDesc**, unsigned short*, char const**, char const**, char const**, unsigned int*) 	security/manager/ssl/nsNSSCallbacks.cpp:329
9 	xul.dll 	mozilla::psm::DoOCSPRequest(std::unique_ptr<PLArenaPool, mozilla::UniquePLArenaPoolDeletePolicy> const&, char const*, mozilla::OriginAttributes const&, SECItemStr const*, mozilla::BaseTimeDuration<mozilla::TimeDurationValueCalculator>, bool, SECItemStr*&) 	security/certverifier/OCSPRequestor.cpp:198
10 	xul.dll 	mozilla::psm::NSSCertDBTrustDomain::CheckRevocation(mozilla::pkix::EndEntityOrCA, mozilla::pkix::CertID const&, mozilla::pkix::Time, mozilla::pkix::Duration, mozilla::pkix::Input const*, mozilla::pkix::Input const*) 	security/certverifier/NSSCertDBTrustDomain.cpp:567
11 	xul.dll 	mozilla::pkix::PathBuildingStep::Check(mozilla::pkix::Input, mozilla::pkix::Input const*, bool&) 	security/pkix/lib/pkixbuild.cpp:241
12 	xul.dll 	mozilla::psm::FindIssuerInner 	security/certverifier/NSSCertDBTrustDomain.cpp:126
13 	xul.dll 	mozilla::psm::NSSCertDBTrustDomain::FindIssuer(mozilla::pkix::Input, mozilla::pkix::TrustDomain::IssuerChecker&, mozilla::pkix::Time) 	security/certverifier/NSSCertDBTrustDomain.cpp:159
14 	xul.dll 	mozilla::pkix::BuildForward 	security/pkix/lib/pkixbuild.cpp:341
15 	xul.dll 	mozilla::pkix::BuildCertChain(mozilla::pkix::TrustDomain&, mozilla::pkix::Input, mozilla::pkix::Time, mozilla::pkix::EndEntityOrCA, mozilla::pkix::KeyUsage, mozilla::pkix::KeyPurposeId, mozilla::pkix::CertPolicyId const&, mozilla::pkix::Input const*) 	security/pkix/lib/pkixbuild.cpp:377
16 	xul.dll 	mozilla::psm::BuildCertChainForOneKeyUsage 	security/certverifier/CertVerifier.cpp:192
17 	xul.dll 	mozilla::psm::CertVerifier::VerifyCert(CERTCertificateStr*, __int64, mozilla::pkix::Time, void*, char const*, std::unique_ptr<CERTCertListStr, mozilla::UniqueCERTCertListDeletePolicy>&, unsigned int, SECItemStr const*, SECItemStr const*, mozilla::OriginAttributes const&, SECOidTag*, mozilla::psm::CertVerifier::OCSPStaplingStatus*, mozilla::psm::KeySizeStatus*, mozilla::psm::SHA1ModeResult*, mozilla::psm::PinningTelemetryInfo*, mozilla::psm::CertificateTransparencyInfo*) 	security/certverifier/CertVerifier.cpp:704
18 	xul.dll 	mozilla::psm::CertVerifier::VerifySSLServerCert(std::unique_ptr<CERTCertificateStr, mozilla::UniqueCERTCertificateDeletePolicy> const&, SECItemStr const*, SECItemStr const*, mozilla::pkix::Time, void*, char const*, std::unique_ptr<CERTCertListStr, mozilla::UniqueCERTCertListDeletePolicy>&, bool, unsigned int, mozilla::OriginAttributes const&, SECOidTag*, mozilla::psm::CertVerifier::OCSPStaplingStatus*, mozilla::psm::KeySizeStatus*, mozilla::psm::SHA1ModeResult*, mozilla::psm::PinningTelemetryInfo*, mozilla::psm::CertificateTransparencyInfo*) 	security/certverifier/CertVerifier.cpp:952
19 	xul.dll 	mozilla::psm::`anonymous namespace'::AuthCertificate 	security/manager/ssl/SSLServerCertVerification.cpp:1422
20 	xul.dll 	mozilla::psm::`anonymous namespace'::SSLServerCertVerificationJob::Run 	security/manager/ssl/SSLServerCertVerification.cpp:1594
21 	xul.dll 	nsThreadPool::Run() 	xpcom/threads/nsThreadPool.cpp:225

The latter looks like some kind of OSCP thing, but the first looks like a generic SSL/TLS problem?  Passing ni? to people who might know more about this than I do.

Comment 3

[Patrick McManus [:mcmanus]]

weirdly the first one looks like it is making progress, but maybe its spinning on something?

david probably would know best.. those are definitely both psm stacks..

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

I haven't had time to chase this down yet, but I'll look at it when I can (which unfortunately might not be soon, given the all-hands next week).

Comment 5

[Marcia Knous [:marcia]]

(In reply to David Keeler [:keeler] (use needinfo?) from comment #4)
> I haven't had time to chase this down yet, but I'll look at it when I can
> (which unfortunately might not be soon, given the all-hands next week).

Hi David - this is hanging around on release as the #11 top browser crash.

Comment 6

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

I think bug 1375709 may have improved this, but maybe I'm just not looking for the right thing in crash-stats. Are there instances of this (or a similar) hang on 56? (or, for that matter, 55?)

Comment 7

[Marcia Knous [:marcia]]

This signature doesn't seem to have crashes in 55 or 56, but is confined to 54. 56 has a few shutdown hangs, but I don't see any similar to this signature. 

Possibly similar crash in 55 would probably be Bug 1377277, but someone with more engineering foo would have to weigh in there.

Comment 8

[Selena Deckelmann :selenamarie :selena]

Keeler - do you think we can close this issue? It doesn't seem to affect anything other than 54.

Comment 9

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Yep - sounds good. Looks like this wasn't bug 1375709 after all, but it appears to have been fixed by something else.