If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

crash in shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread

NEW
Unassigned

Status

()

Core
Security: PSM
P3
critical
2 years ago
19 days ago

People

(Reporter: khuey, Unassigned)

Tracking

({hang, topcrash})

unspecified
x86
Windows NT
hang, topcrash
Points:
---

Firefox Tracking Flags

(firefox47 affected, firefox48 affected, firefox49 affected, firefox-esr45 affected, firefox50 affected, firefox51 affected, firefox-esr52 affected, firefox55 affected, firefox56 affected, firefox57 affected, firefox58 affected)

Details

(Whiteboard: [psm-smartcard][tbird crash], crash signature)

This bug was filed from the Socorro interface and is 
report bp-b774fd7e-eaa8-44e4-b9e5-187152160210.
=============================================================

Shutdown hang on the smart card thread.  The main thread is shutting down NSS and waiting for the smart card thread to join.  The smart card thread (thread 26 in this report) is doing:

0 	KERNELBASE.dll 	TlsGetValue 	
1 	nss3.dll 	PR_Lock 	nsprpub/pr/src/threads/combined/prulock.c
2 	nss3.dll 	nsslist_add_element 	security/nss/lib/base/list.c
3 	nss3.dll 	nssList_Add 	security/nss/lib/base/list.c
4 	nss3.dll 	nssList_Clone 	security/nss/lib/base/list.c
5 	nss3.dll 	nssList_CreateIterator 	security/nss/lib/base/list.c
6 	nss3.dll 	STAN_ResetTokenInterator 	security/nss/lib/pki/pki3hack.c
7 	nss3.dll 	SECMOD_UpdateSlotList 	security/nss/lib/pk11wrap/pk11util.c
8 	nss3.dll 	secmod_HandleWaitForSlotEvent 	security/nss/lib/pk11wrap/pk11util.c
9 	nss3.dll 	SECMOD_WaitForAnyTokenEvent 	security/nss/lib/pk11wrap/pk11util.c
10 	xul.dll 	SmartCardMonitoringThread::Execute() 	security/manager/ssl/nsSmartCardMonitor.cpp
11 	xul.dll 	SmartCardMonitoringThread::LaunchExecute(void*) 	security/manager/ssl/nsSmartCardMonitor.cpp
Flags: needinfo?(rlb)
Flags: needinfo?(rlb) → needinfo?(dkeeler)
Looks like onepin-opensc-pkcs11.dll has been loaded in that profile. I've already found one way in which it's broken for NSS/Firefox, but that bug doesn't result in the same symptoms for me, so I imagine there are other issues as well.
(As a temporary workaround, you can remove that security module in about:preferences -> Advanced -> Certificates -> Security Devices.)
Flags: needinfo?(dkeeler)
This isn't actually my crash report, it's something I found from Socorro's topcrash list.
I think this is due to https://github.com/OpenSC/OpenSC/issues/683 (In debug builds, you get an assertion failure. In opt builds, I'm seeing a shutdown hang.)
This is partly the fault of NSS in that it doesn't handle PKCS11 modules that don't behave quite the way we're expecting, but it also does appear to be a bug in OpenSC. It would be nice if we could somehow sandbox these modules (maybe via a shim module that loads these modules in a separate process and can enforce more stringent checks? Also maybe NSS just needs to do more error checking/handling.)
Whiteboard: [psm-smartcard]
Crash volume for signature 'shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread':
 - aurora  (version 49): 37 crashes from 2016-06-07.
 - beta    (version 48): 1262 crashes from 2016-06-06.
 - release (version 47): 13619 crashes from 2016-05-31.
 - esr     (version 45): 2532 crashes from 2016-04-07.

Crash volume on the last weeks:
             Week N-1   Week N-2   Week N-3   Week N-4   Week N-5   Week N-6   Week N-7
 - nightly          0          0          0          0          0          0          0
 - aurora           1          3          8          0          0         19          5
 - beta           184        167        198        183        195        207         58
 - release       2144       2037       2096       2203       1840       1939        500
 - esr            299        308        339        297        261        315        196

Affected platform: Windows
status-firefox47: --- → affected
status-firefox48: --- → affected
status-firefox49: --- → affected
status-firefox-esr45: --- → affected
Crash volume for signature 'shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread':
 - nightly(version 50):3 crashes from 2016-06-06.
 - aurora (version 49):37 crashes from 2016-06-07.
 - beta   (version 48):1383 crashes from 2016-06-06.
 - release(version 47):15046 crashes from 2016-05-31.
 - esr    (version 45):2767 crashes from 2016-04-07.

Crash volume on the last weeks:
            W. N-1  W. N-2  W. N-3  W. N-4  W. N-5  W. N-6  W. N-7
 - nightly       2       0       0       0       0       0       0
 - aurora        1       1       3       8       0       0      19
 - beta        187     184     167     198     183     195     207
 - release    2246    2144    2037    2096    2203    1840    1939
 - esr         371     299     308     339     297     261     315

Affected platform: Windows
status-firefox50: --- → affected
Crash volume for signature 'shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread':
 - nightly (version 51): 3 crashes from 2016-08-01.
 - aurora  (version 50): 33 crashes from 2016-08-01.
 - beta    (version 49): 254 crashes from 2016-08-02.
 - release (version 48): 1173 crashes from 2016-07-25.
 - esr     (version 45): 3279 crashes from 2016-05-02.

Crash volume on the last weeks (Week N is from 08-22 to 08-28):
            W. N-1  W. N-2  W. N-3
 - nightly       1       1       1
 - aurora        9      15       9
 - beta         71     140      43
 - release     274     620     279
 - esr         139     291     376

Affected platform: Windows

Crash rank on the last 7 days:
             Browser   Content   Plugin
 - nightly
 - aurora
 - beta
 - release
 - esr
status-firefox51: --- → affected
The 2nd signature alone is currently at #28 in the Nightly Top Crash List.
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] → [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] [@ shutdownhang | PR_JoinThread | SmartCardMonitoringThread::`scalar deleting destructor'' ]
Keywords: topcrash
Crash Signature: [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] [@ shutdownhang | PR_JoinThread | SmartCardMonitoringThread::`scalar deleting destructor'' ] → [@ shutdownhang | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] [@ sh…

Updated

11 months ago
Whiteboard: [psm-smartcard] → [psm-smartcard][tbird crash]

Comment 8

8 months ago
(In reply to Mats Palmgren (:mats) from comment #7)
> The 2nd signature alone is currently at #28 in the Nightly Top Crash List.

perhaps a blip, because it is no longer in the top 50.
However, it is #13 for Firefox 51.0.1.

N.B. 40% of crashs are pt-BR locale. I've seen high rate of security-related crashes with pt-BR for other signatures. Also sometimes for "it" locale.
https://crash-stats.mozilla.com/search/?signature=%3Dshutdownhang%20%7C%20PR_JoinThread%20%7C%20SmartCardMonitoringThread%3A%3A~SmartCardMonitoringThread&date=%3E%3D2017-02-06T13%3A48%3A00.000Z&date=%3C2017-02-13T13%3A48%3A00.000Z&_sort=-date&_facets=signature&_facets=useragent_locale&_columns=date&_columns=product&_columns=version&_columns=platform#facet-useragent_locale

This Thunderbird user ("it" locale) sometimes crashes several times a day bp-6710ade1-76de-4f0e-acb4-5ac562170213  shutdownhang | _PR_MD_WAIT_CV | _PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::Stop

Comment 9

7 months ago
FYI and hopefully further progress. I'm was issued a SC based signature dongle to use for signing docs and logging in to various websites. Ever since that was installed, FF (all versions) never closes appropriately and end up crashing upon closure. 

The USB dongle is a "StarSign Crypto USB Token S" made by "Giesecke & Devrient GmbH".
The drivers and docs I installed are:
https://www.gi-de.com/gd_media/media/en/documents/brochures/mobile_security_2/nb/StarSign_Crypto_USB_Token.pdf
http://www.elektroninis.lt/bylos/elektroninis_lt/Diegliai/Elektroninis-lt-kompiuterio-paruosimas-GD-USB-laikmenai.exe

Somewhere along the line (AFAICR) I managed to intercept the installer which ended up with installing two other programs: 

TokenManager.exe
firefoxinstaller.exe

The "firefoxinstaller" is made by: "A.E.T. Europe B.V" and installs/depends on: libcastella.dll and
aetpkss1.dll among others.

Updated

6 months ago
Crash Signature: [@ shutdownhang | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] [@ sh… → [@ shutdownhang | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] [@ shutdownhang | WaitForSingleObjectEx | WaitForSingleObject | PR_WaitCondVar | PR_JoinThread | SmartCardMonitoringThread::~SmartCardMonitoringThread] [@ sh…

Updated

4 months ago
See Also: → bug 1375344
Priority: -- → P3

Comment 10

19 days ago
https://crash-stats.mozilla.com/search/?signature=%5Eshutdownhang&submitted_from_infobar=%21__true__&product=Firefox&date=%3E%3D2017-09-27T18%3A00%3A00.000Z&date=%3C2017-09-28T11%3A36%3A00.000Z&_sort=-date&_facets=signature&_facets=release_channel&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature

Combining signatures, this is probably the 2nd or 3rd top browser shutdown hang we have. Is there anything we can do here?
Flags: needinfo?(dkeeler)

Updated

19 days ago
status-firefox55: --- → affected
status-firefox56: --- → affected
status-firefox57: --- → affected
status-firefox58: --- → affected
status-firefox-esr52: --- → affected
If a user has a smart card (PKCS#11 device) with removable slots, Firefox will launch a thread for each module and loop calling SECMOD_WaitForAnyTokenEvent to be alerted to any insertions/removals. At shutdown, we call SECMOD_CancelWait, which should cancel any waiting threads. However, since that involves calling 3rd party code, we really have no idea if these modules are behaving correctly. The real solution is to stop relying on PKCS#11, but since that's unlikely in the near future, the next best thing would be to load these modules in a child process. That way, misbehaving modules don't cause Firefox to hang/crash/etc. That's a lot of engineering work, though. It may be possible to avoid this particular issue if we never call SECMOD_WaitForAnyTokenEvent. I filed bug 1381154 but it didn't look promising at the time. Since this is such a common issue, though, I'll have another go at making that happen.
Flags: needinfo?(dkeeler)
You need to log in before you can comment on or make changes to this bug.