1873693 - Crash in [@ shutdownhang | builtins_static::pkcs11::get_search_cache]

Status: RESOLVED INCOMPLETE

(Core :: Security: PSM, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Reported as 181 crashes from 165 installs, all on Windows with 64-bit builds of Firefox 121.0 or 121.0rc1. No reports for nightly or beta builds or earlier (or later, but that's expected if we didn't see it for development builds of v121)) versions.

Crash report: https://crash-stats.mozilla.org/report/index/ac9180c7-95f6-4859-ac57-87b880240109

MOZ_CRASH Reason: Shutdown hanging at step XPCOMShutdown. Something is blocking the main-thread.

Top 10 frames of crashing thread:

0  nssckbi.dll  enum2$<core::option::Option<alloc::collections::btree::map::BTreeMap<u32, smallvec::SmallVec<array$<builtins_static::internal::ObjectHandle, 1> >, alloc::alloc::Global> > >::as_mut  /rustc/cc66ad468955717ab92600c770da8c1601a4ff33/library/core/src/option.rs:696
0  nssckbi.dll  builtins_static::pkcs11::get_search_cache  security/manager/ssl/builtins/src/pkcs11.rs:102
0  nssckbi.dll  builtins_static::pkcs11::close_all_sessions  security/manager/ssl/builtins/src/pkcs11.rs:204
0  nssckbi.dll  builtins_static::pkcs11::C_CloseAllSessions  security/manager/ssl/builtins/src/pkcs11.rs:453
1  nss3.dll  PK11_DestroySlot  security/nss/lib/pk11wrap/pk11slot.c:453
2  nss3.dll  PK11_FreeSlot  security/nss/lib/pk11wrap/pk11slot.c:490
2  nss3.dll  SECMOD_DestroyModule  security/nss/lib/pk11wrap/pk11util.c:904
3  nss3.dll  SECMOD_DestroyModuleListElement  security/nss/lib/pk11wrap/pk11util.c:950
3  nss3.dll  SECMOD_DestroyModuleList  security/nss/lib/pk11wrap/pk11util.c:965
3  nss3.dll  SECMOD_Shutdown  security/nss/lib/pk11wrap/pk11util.c:68

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:keeler, could you have a look please?

For more information, please visit BugBot documentation.

Comment 2

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

I'm baffled as to how we could be hanging there. Any ideas, John?

Comment 3

[John Schanck [:jschanck]]

Maybe the socket thread, or an SSL cert thread, was cancelled while holding the same mutex? I don't actually know whether we cancel threads at shutdown, or how we terminate long-running non-main threads, but the socket and ssl cert threads are not present in the crash reports. It is (maybe?) interesting that the osclientcert thread is still alive though.

Comment 4

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

My understanding is we cancel threads cooperatively (as in, we don't just kill them), so the mutex shouldn't be held. In any case, if I'm reading the stack correctly, we're not waiting on a mutex - we're calling Option::as_mut().

It is odd that the osclientcert thread is still alive, but I don't really see how that could affect this.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Since the crash volume is low (less than 15 per week), the severity is downgraded to S3. Feel free to change it back if you think the bug is still critical.

For more information, please visit BugBot documentation.

Comment 6

[John Schanck [:jschanck]]

This seems to have only affected one build (buildid=20231211174248). I suspect the stack we see here is not actually the reason for the crashes.