1551177 - skip self-signed intermediate certificates that aren't trust anchors in path building

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P1)

Description

[stigmh]

Attachment: fftls.tar.gz

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0

Steps to reproduce:

Video: https://youtu.be/Gs6rdhW-wSM

1. Configure NGINX server with self-signed certificate and self-generated Diffie-Hellman. This is done through Docker and the attached script.

2. Visit server through HTTPS

3. Cancel the docker image with Ctrl+C; Script deletes and regenerates self-signed certificate, DH, and start new Docker instance.

Repeat 2-3 until complete hang. Same issue happens without server side Docker and on a Fedora client.

Actual results:

Firefox eventually fails to load the server's page claiming it is "Performing TLS handshake"; it hangs. I have to kill Firefox through process manager. Seems like cert9.db in profile catalog somehow becomes corrupt as deleting it resolves the issue.

Expected results:

Page should load normally; no weird delay or hang. Brave (Chromium) and MS Edge are not affected.

Comment 1

[Dana Keeler (she/her) (use needinfo) [:keeler]]

As a workaround you can clear the certificates that have accumulated in your certificate database (cert9.db) either using certutil or by just deleting the file.

The solution from bug 1056341 isn't flexible enough to really work given the wide range in how powerful peoples' devices are.

Comment 2

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: Bug 1551177 - avoid searching unproductive certificate paths during verification r?jcj!,KevinJacobs!

In bug 1056341 we introduced a search budget to mozilla::pkix to attempt to work
around the problem of having an extremely large search space given a set of
certificates all with the same subject and issuer distinguished names but
different public keys. In the end, though, there is probably no good value to
choose for the budget that is small enough to run quickly on the wide range of
hardware our users have and yet is large enough that we're confident won't break
someone's complicated pki setup (looking at you, the US federal government).

To address this, use the observation that as long as an intermediate can't add
information necessary to build a certificate chain (e.g. stapled SCTs), we
should never need a self-signed intermediate (as in, its own key verifies the
signature on it and its subject and issuer distinguished names are identical) to
build a trusted chain (since the exact same chain without that intermediate
should be valid). Given this, we simply skip all self-signed non-trust anchor
CA certificates during path building.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5a721a7648f2
avoid searching unproductive certificate paths during verification r=jcj,KevinJacobs

Comment 4

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/5a721a7648f2

Comment 6

[Steven Singer]

The fix for this bug seems to be implicated in a major network performance regression I'm seeing on Firefox Nightly. See bug 1577944.

Comment 7

[Steven Singer]

Comment 2 of 1577944 has more detail, but the summary is the problem is the apparently innocuous call to NSSCertDBTrustDomain::GetCertTrust(). This makes a call to CERT_NewTempCertificate() on the assumption that it will be fast because the certificate will be in cache. But, profiling shows that, on my system at least, this is not the case (perhaps because it's an enterprise certificate?). When the cache is missed, this call is very expensive and triggers a lock convoy. Speculation in bug 1555306, comment 9 suggests that the code may do I/O access (with 100 ms busy waits) while holding the lock.

Comment 8

[nemo]

Oh, another bug to link to this one perhaps. Someone just bisected to this commit in bug #1570222 which pretty much broke all the dev servers/management consoles at work using self-signed certs, forcing almost all the network admins/dev teams onto IE/Chrome…