2069597 - Startup hang on macOS: NSS deadlock in SSL Cert threads after cold launch with a VPN interface up; persists until an intact key4.db is removed

Status: NEW

(Thunderbird :: Security, defect)

Description

[Bart[:mormegil]]

Attachment: thunderbird_sample.txt

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.0.0 Safari/537.36

Steps to reproduce:

1. Have a Thunderbird profile whose credential-store files, cert9.db, key4.db, logins.json, logins-backup.json, and logins.db, are corrupted. The exact corruption trigger is unknown; the profile reached this state through normal day-to-day use, with no unusual event identified.
2. Launch Thunderbird normally.

Actual results:

Thunderbird's window appears but startup never completes, producing an indefinite hang that requires a force-quit. This reproduced with all add-ons disabled in Safe Mode and with the network fully offline, which rules out extensions and live network or OAuth calls as the cause.

A live stack sample (sample <pid>, about 3.4 seconds, 3371 samples) shows the main thread permanently parked in a timed condition-variable wait during startup, never advancing:

Thread_1000968 DispatchQueue_1: com.apple.main-thread (serial)
  ... (XUL frames, symbols stripped in release build) ...
  mozilla::detail::ConditionVariableImpl::wait_for(...)
  _pthread_cond_wait
  __psynch_cvwait

At the same time, two NSS (Network Security Services) worker threads, "SSL Cert #1" and "SSL Cert #2" in libnss3.dylib, are permanently parked in an unconditional wait on the same class of condition variable in 100 percent of samples, never completing:

Thread_1001239: SSL Cert #1 / Thread_1001375: SSL Cert #2
  ... (libnss3.dylib / XUL frames, symbols stripped) ...
  mozilla::detail::ConditionVariableImpl::wait(...)
  _pthread_cond_wait
  __psynch_cvwait

This points at the NSS certificate-verification worker pool never finishing whatever operation it was given against the corrupted cert9.db and key4.db, while the main thread's startup path waits synchronously on that work. The result is an indefinite hang, without any error message or timeout.

Moving cert9.db, key4.db, logins.json, logins-backup.json, and logins.db out of the profile directory and letting Thunderbird regenerate them fixed the hang immediately and permanently.

Environment: macOS 15.7.9 (24G830), Apple Silicon (arm64), Thunderbird 153.2.0 (15326.8.28).

This may be the same general class of indefinite hang at macOS startup as bug 2060146, but that bug's proposed cause, Adobe Creative Cloud font retrieval, does not apply here. Creative Cloud is not installed on this machine, and the hang reproduced with all third-party software and networking disabled, which points at a distinct root cause.

Expected results:

Thunderbird should start normally, or, if the credential or certificate store is unreadable or corrupted, detect that condition and recover, for example by regenerating the store or surfacing an error, rather than blocking startup indefinitely without any feedback to the user.

Comment 1

[Bart[:mormegil]]

Attachment: sample output: startup deadlock reproduced on demand (VPN up, cold launch)

Captured from a deliberately reproduced hang rather than a wild occurrence: tailscale up, then a cold launch of Thunderbird. Same signature as attachment 9637027 [details], which came from the original 4 September occurrence. Keeping both so the reproduction can be checked against the case that happened unprompted.

Comment 2

[Bart[:mormegil]]

Correcting this report: the credential files are not corrupt, and the cause I originally gave was wrong.

I filed this saying the hang was "caused by corrupted profile credential-store files". That was an inference from the workaround working, not something I had measured, and it does not survive testing. Apologies for sending this in the wrong direction. I now have a reproduction recipe and a bisect, both below.

Reproduction (reproduced on first attempt, then repeatedly):

1. Quit Thunderbird completely.
2. Bring a VPN up that creates a utun interface. I used tailscale up (Tailscale 1.102.3).
3. Launch Thunderbird cold.

It deadlocks during startup initialisation and never presents a window.

Stack signature, from sample, 3495 of 3495 samples on each thread:

Thread_x  DispatchQueue_1: com.apple.main-thread
  ...
  mozilla::detail::ConditionVariableImpl::wait_for(...)   (in libmozglue.dylib)
  __psynch_cvwait

Thread_y: SSL Cert #1
  ??? (in libnss3.dylib) load address ... + 0x1a5fbc
  ...
  mozilla::detail::ConditionVariableImpl::wait(...)       (in libmozglue.dylib)

Note the asymmetry: the main thread is in a timed wait, the NSS worker is in an unconditional one. For contrast, a healthy run of the same profile has the main thread in mach_msg and the SSL Cert worker in wait_for.

The trigger is startup, not runtime. I first tried toggling the VPN against an already-running, settled Thunderbird holding 14 live IMAP connections: 8 up/down cycles, 16 transitions, sampling after every one. Zero hits, the signature stayed at 0 blocked SSL Cert threads throughout. Only a cold launch with the interface already up reproduces it.

Once triggered it persists across restarts, regardless of VPN state. After the initial hang I stopped Tailscale and relaunched without touching any file: still deadlocked. This is why restarting never helped on my two earlier occurrences.

Bisect: key4.db is the carrier, and nothing else is.

action	SSL Cert threads in unconditional wait
remove cert9.db only	2, still deadlocked
also remove key4.db	0, starts normally
restore that same key4.db	2, deadlocked again

So the workaround in the whiteboard is correct but over-broad. Only key4.db needs to be removed.

Nothing is corrupt, and I checked rather than assumed:

• PRAGMA integrity_check returns ok on cert9.db, key4.db and logins.db, in both of my earlier failing sets.
• logins.json and logins-backup.json both parse as valid JSON.
• freelist_count 0, journal_mode delete, no stale journal or WAL files alongside them.
• key4.db's metaData carries the same key structure as a working store.
• lsof on the hung process shows cert9.db, key4.db and logins.db all open read/write, so opening them had already succeeded.

The key4.db that deadlocks is byte-identical to one that worked. I snapshotted the credential set while the profile was running healthily, triggered the hang about 20 minutes later, and cmp reports cert9.db, key4.db and logins.db unchanged. The same bytes that had been serving a running Thunderbird for roughly 24 hours are now deterministically fatal at startup.

Store contents appear irrelevant to whether it happens. My first occurrence had 111 rows in nssPublic and 2 in nssPrivate. My second had 3 and 1, in a store Thunderbird had created itself five days earlier. Both deadlock identically, so this is not about accumulated content.

Open question I cannot answer from here: if key4.db is unchanged and structurally sound, something outside the file has to be pairing with it to make the same bytes fatal. I have not identified what. That seems like the thread worth pulling.

The sample output from this reproduction is attached (attachment 9640550 [details]). I also still have the failing key4.db intact, along with a known-good snapshot of the same profile taken shortly before the trigger. I have not attached those, since key4.db is a private key store -- happy to provide them through whatever channel you prefer. I can trigger this on demand, so I am glad to run specific tests against it.

Environment: Thunderbird 153.2.0, macOS 15.7.9 (24G830), arm64, Tailscale 1.102.3, 9 utun interfaces present.