1854653 - Crash in [@ core::result::unwrap_failed | neqo_crypto::init::closure$0]

Status: RESOLVED WORKSFORME

(Core :: Networking: HTTP, defect, P2)

Description

[Randell Jesup [:jesup] (needinfo me)]

Crash initializing neqo_crypto, with one of two errors:
NSS_NoDB_Init failed: NssError { name: "SEC_ERROR_PKCS11_DEVICE_ERROR", code: -8023, desc: "A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot." }
or
NSS_NoDB_Init failed: NssError { name: "PR_CALL_ONCE_ERROR", code: -5925, desc: "The one-time function was previously called and failed. Its error code is no longer available" }

Goes back to at least 111, but seems to have spiked some since July.

Crash report: https://crash-stats.mozilla.org/report/index/129d60d4-f4ec-47de-9f39-d65030230922

MOZ_CRASH Reason: NSS_NoDB_Init failed: NssError { name: "PR_CALL_ONCE_ERROR", code: -5925, desc: "The one-time function was previously called and failed. Its error code is no longer available" }

Top 10 frames of crashing thread:

0  xul.dll  MOZ_Crash  mfbt/Assertions.h:281
0  xul.dll  RustMozCrash  mozglue/static/rust/wrappers.cpp:17
1  xul.dll  mozglue_static::panic_hook  mozglue/static/rust/lib.rs:96
2  xul.dll  core::ops::function::Fn::call<void   /rustc/8ede3aae28fe6e4d52b38157d7bfe0d3bceef225/library/core/src/ops/function.rs:250
3  xul.dll  alloc::boxed::impl$49::call  library/alloc/src/boxed.rs:1999
3  xul.dll  std::panicking::rust_panic_with_hook  library/std/src/panicking.rs:709
4  xul.dll  std::panicking::begin_panic_handler::closure$0  library/std/src/panicking.rs:597
5  xul.dll  std::sys_common::backtrace::__rust_end_short_backtrace<std::panicking::begin_panic_handler::closure_env$0, never$>  library/std/src/sys_common/backtrace.rs:151
6  xul.dll  std::panicking::begin_panic_handler  library/std/src/panicking.rs:593
7  xul.dll  core::panicking::panic_fmt  library/core/src/panicking.rs:67

Comment 1

[Kershaw Chang [:kershaw]]

Maybe we should not crash, but just fallback to h2.

Comment 2

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

I think we can make neqo_crypto::init not panic and return a Result instead, right?
Assuming we can't do anything about the NSS error?
And the question is whether H2 crypto with nss will work then?

Comment 3

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

I had a chat with Martin about it, and most likely there's not much we can do about it.
We can keep investigating, but it's probably low return on investment. It's also likely the crashes are only happening for certain users and not affecting the majority of the population.

Comment 4

[Martin Thomson [:mt:]]

Just to expand on this: The approach we take for initializing NSS is close to the absolute minimum we could do. When Firefox initializes NSS, it has several fallbacks. This approach is the option of last resort there. For instance, we do not use a database or external PKCS#11 modules here.

Two possible things with low likelihood of positive results:

1. There are some flags that we could maybe tweak to make this more permissive. However, those do not relate to what is showing up in the crash stats.
2. We could just try a couple of times. Transient errors do occur and we might avoid them for longer if we try multiple times. However, if the true cause is file or memory corruption, we're just shifting the point at which the crash occurs.

Neither approach seems great. We need NSS to do QUIC. Crashing when we can't get it is an entirely reasonable thing to do.

Comment 5

[Martin Thomson [:mt:]]

Randell's change to the signature indicates a new class of problem that might even warrant a different bug number. That signature indicates that the version check failed, which is a different problem to what we were originally considering.

We don't always ship NSS with Firefox. If we happen to land on a system with an old NSS, then this is what happens.

Comment 6

[Lars Eggert [:lars]]

I'm surprised we don't ship NSS with Firefox, given that neqo really has hard dependencies on NSS versions.

What would a good remedy here be? Simply pass up the error instead of asserting? I did that in https://github.com/mozilla/neqo/pull/2030

Comment 7

[Martin Thomson [:mt:]]

For various reasons, NSS is bundled separately in some distributions. In some cases, this is because NSS is subject to system-wide policy about the availability of certain capabilities. This includes FIPS certification.

However, system integrators who make this choice are responsible for ensuring that the Firefox packet has the correct dependency on NSS. It should not be possible to install Firefox without upgrading NSS.

One way we might deal with this is to run the neqo init and disable QUIC if it fails.

Comment 8

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

The latest crash I can find is 127.0.2