1674410 - Shutdown of the SSL Cert threadpool hangs

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P3)

Description

[Jens Stutte [:jstutte]]

From bug 1633342 comment 41:

Looking at shutdownhang | mozilla::TaskController::GetRunnableForMTTask | nsThread::Shutdown | mozilla::net::nsSocketTransportService::ShutdownThread.

In all the reports I clicked on, the SocketThread is stuck while shutting down the SSL Cert threadpool.

Looking at the shutdown function, it seems we shutdown the threads in the order we created them (and wait for each single thread before we loop). I see in the first three reports I clicked on, that SSL Cert #1 is still alive, and am assuming that it processes some long lasting event when the shutdown event comes in, such that we never get to process the shutdown event.

In two cases it is stuck in nsNSSComponent::BlockUntilLoadableCertsLoaded(), in the other case mozilla::psm::NSSCertDBTrustDomain::GetCertTrust seems stuck.

We might want to consider the use of ShutdownWithTimeout in StopSSLServerCertVerificationThreads ?

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:keeler, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 2

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

There's a couple of things going on here. First is third-party PKCS#11 modules. As long as we support loading third-party modules, they'll continue to give users a bad experience. Our hope is that osclientcerts will make it so users don't have to load these modules, but we're not quite there yet. The other issue is NSS being essentially not thread-safe. Any time we use NSS resources on these other threads, we could hit a deadlock/race condition/etc., so bug 1664048 is working on avoiding NSS types in PSM. There will be some cases where NSS types can't be avoided, so we'll have to do something like proxy that work to the socket thread (since the socket thread can't not use NSS resources as well, since that's how we TLS).

Comment 3

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Oh also we can't not shut down those threads because then when we shut NSS down, it'll fail because those resources are still in use.

Comment 4

[Gabriele Svelto [:gsvelto] (PTO)]

Adjusting the signature in preparation for bug 1794587.

Comment 5

[Dana Keeler (she/her) (use needinfo) [:keeler] (on leave)]

Attachment: Bug 1674410 - stop slow certificate verification tasks when the app is shutting down r?jschanck

Certificate verification can take a while, which is why it runs in a separate
thread pool. At shutdown, the thread pool gets joined. To make this fast,
certificate verification tasks should check for shutdown before doing
time-consuming operations and return early if appropriate.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6c8edd3b4ff7
stop slow certificate verification tasks when the app is shutting down r=jschanck

Comment 7

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/6c8edd3b4ff7