1958140 - Firefox 137 cannot resolve any URL on MacOS 15.4 (with thousands of keys in the Mac keychain)

Status: RESOLVED FIXED

(Core :: Security: PSM, defect, P1)

Description

[Gavin Heverly-Coulson]

Attachment: Pending Crash Report generated after browser crashed while quitting

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36

Steps to reproduce:

After updating to Firefox 137, I have not been able to resolve and load any URL in Firefox. Additionally (not sure if it's connected), when quitting the browser it hangs for up to a minute before crashing.

I have tested resetting the profile on Firefox and a clean install (removing all Firefox and Mozilla data in ~/Library/Application Support/, ~/Library/Preferences/, and ~/Library/Caches/) with the same result in all cases.

Downgrading to Firefox 136.0.4 works as expected, can navigate to any website I choose, browser quits cleanly without crashing.

Computer specs:
MacOS 15.4
Macbook Pro 14" 2021 (M1 Max)
User-agent, since I cannot file this from the not working Firefox version - "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:137.0) Gecko/20100101 Firefox/137.0"

The crash reporter also appears to not being able to connect to its server, since I have multiple pending reports from today. I have attached the .dmp and .extra files from one of them to this report.

Actual results:

Other browsers (Safari, Chrome) and applications (Terminal, Slack, etc) can load data without issue. I have tested this on three different networks (Home, Office, Office Guest) and see the same issue on all three.

As an observation, on the default home screen, the weather widget can identify my location and display what appears to be current, accurate weather. The "Thought-provoking Stories" section loads headlines but does not show any thumbnail images for the stories, so it appears that something within the system is able to resolve data over the network.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Networking' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Kershaw Chang [:kershaw]]

Hi Reporter,

Thanks for reporting this.
Could you try to capture a http log?
In about:logging, please select logging to file and send the file to [e-mail].

Thanks.

Comment 3

[Gavin Heverly-Coulson]

Hi Kershaw, I've captured the log and emailed it as requested. Thanks for looking into this.

Comment 4

[Tim Small]

Running into the same issue too on a MacBook Pro (with M2 Pro) running macOS 15.3.2, but I was able to pinpoint down that it has something to do with OS certificates. I initially noticed when Firefox updated to v137.0.0, but it's still persisting with the recently released v137.0.1.

When I tried to open the View certificates panel on about:preferences#privacy, it completely froze the browser. So I did some digging in about:config and set security.osclientcerts.autoload to false. Once I completely relaunched Firefox, I was able to connect to websites without issue.

If you need me to collect any logs or anything, just let me know what you need and I'll be happy send them your way.

Comment 5

[Kershaw Chang [:kershaw]]

(In reply to Tim Small from comment #4)

Running into the same issue too on a MacBook Pro (with M2 Pro) running macOS 15.3.2, but I was able to pinpoint down that it has something to do with OS certificates. I initially noticed when Firefox updated to v137.0.0, but it's still persisting with the recently released v137.0.1.

When I tried to open the View certificates panel on about:preferences#privacy, it completely froze the browser. So I did some digging in about:config and set security.osclientcerts.autoload to false. Once I completely relaunched Firefox, I was able to connect to websites without issue.

If you need me to collect any logs or anything, just let me know what you need and I'll be happy send them your way.

Yes, please also capture a http log.
Thank you.

Comment 6

[Kershaw Chang [:kershaw]]

(In reply to Gavin Heverly-Coulson from comment #3)

Hi Kershaw, I've captured the log and emailed it as requested. Thanks for looking into this.

Thanks for the log.
It looks like the socket thread was blocked by something — all the activity I see on the socket thread is connections being closed due to timeout.

2025-04-07 16:04:03.185005 UTC - [Parent 23329: Socket Thread]: V/nsHttp canceling transaction: tls handshake takes too long: tls handshake last 212211ms, timeout is 30000ms.
2025-04-07 16:04:03.185008 UTC - [Parent 23329: Socket Thread]: V/nsHttp nsHttpConnection::CloseTransaction[this=1538acd00 trans=132fddde0 reason=804b000e]
2025-04-07 16:04:03.185012 UTC - [Parent 23329: Socket Thread]: V/nsHttp   closing associated mTransaction
2025-04-07 16:04:03.185015 UTC - [Parent 23329: Socket Thread]: V/nsHttp SpeculativeTransaction::Close 132fddde0 aReason=804b000e
...
2025-04-07 16:04:13.603974 UTC - [Parent 23329: Socket Thread]: V/nsHttp canceling transaction: tls handshake takes too long: tls handshake last 263844ms, timeout is 30000ms.
2025-04-07 16:04:13.603978 UTC - [Parent 23329: Socket Thread]: V/nsHttp nsHttpConnection::CloseTransaction[this=1538ad900 trans=132fdeda0 reason=804b000e]

I'm not sure if this is related to security.osclientcerts.autoload, as mentioned in comment #4.

Dana, could you take a look? Thanks!

Comment 7

[Dana Keeler (she/her) (use needinfo) [:keeler]]

In Firefox 138 (this doesn't work yet in 137) with security.osclientcerts.autoload set to true, can you enable the profiler (https://profiler.firefox.com/) and capture a profile with the custom threads set to "socket,ssl,osclientcerts"?

Comment 8

[Tim Small]

(In reply to Dana Keeler (she/her) (use needinfo) [:keeler] from comment #7)

In Firefox 138 (this doesn't work yet in 137) with security.osclientcerts.autoload set to true, can you enable the profiler (https://profiler.firefox.com/) and capture a profile with the custom threads set to "socket,ssl,osclientcerts"?

I've sent a profile generated for that. I had to set security.osclientcerts.autoload to false at the end of the capture in order for me to be able to collect it (Without doing that, it would cause Firefox to hang).

Comment 9

[Tim Small]

Actually, disregard the first one I sent. Loaded the profile the wrong way, so it didn't include the socket,ssl,osclientcerts threads.

Comment 10

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Instead of sending it to necko@, can you send it to me? (I'm not cc'd on that email)

Comment 11

[Tim Small]

Ah! Sorry about that! Sent it to you directly.

Comment 12

[Gavin Heverly-Coulson]

(In reply to Dana Keeler (she/her) (use needinfo) [:keeler] from comment #7)

In Firefox 138 (this doesn't work yet in 137) with security.osclientcerts.autoload set to true, can you enable the profiler (https://profiler.firefox.com/) and capture a profile with the custom threads set to "socket,ssl,osclientcerts"?

I've sent the requested profiling result to your email, let me know if you need anything else from me.

Comment 13

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Where we're at here is bug 1959526 should help a lot. Additionally, what we're seeing is having a keychain with thousands of keys can slow things down considerably. It's unclear that there's anything Firefox would be able to do to work around that, unfortunately.

Comment 14

[Gavin Heverly-Coulson]

(In reply to Dana Keeler (she/her) (use needinfo) [:keeler] from comment #13)

Additionally, what we're seeing is having a keychain with thousands of keys can slow things down considerably. It's unclear that there's anything Firefox would be able to do to work around that, unfortunately.

I can confirm that this has addressed my issue. I had over 67,000 private keys in my keychain, turned out 66,500 of them were from an old, now resolved, VPN bug that duplicated keys. After deleting those, Firefox 137 is working for me.

Comment 15

[Tim Small]

(In reply to Gavin Heverly-Coulson from comment #14)

I can confirm that this has addressed my issue. I had over 67,000 private keys in my keychain, turned out 66,500 of them were from an old, now resolved, VPN bug that duplicated keys. After deleting those, Firefox 137 is working for me.

What VPN software caused the duplicate keys? I've been trying to pinpoint down why I had over 18,000 random private keys.

(In reply to Dana Keeler (she/her) (use needinfo) [:keeler] from comment #13)

Where we're at here is bug 1959526 should help a lot. Additionally, what we're seeing is having a keychain with thousands of keys can slow things down considerably. It's unclear that there's anything Firefox would be able to do to work around that, unfortunately.

I haven't had a chance to look over the Rust code again this week, but does the call to SecItemCopyMatching just list all items in the keychain or does it do a specific query? No idea if that's helpful in any way. I also noticed Apple specifies it's a blocking call, which isn't really helpful either.

Comment 16

[Brandon]

FWIW I cleaned up my Keychain, and that fixed Firefox. I wrote about it here:

https://connect.mozilla.org/t5/discussions/firefox-137-hangs-and-will-not-resolve-most-sites-now/m-p/93350#M36345

Comment 17

[Kershaw Chang [:kershaw]]

Change the component to PSM, since this is not a networking issue.

Comment 18

[Gavin Heverly-Coulson]

(In reply to Tim Small from comment #15)

(In reply to Gavin Heverly-Coulson from comment #14)

I can confirm that this has addressed my issue. I had over 67,000 private keys in my keychain, turned out 66,500 of them were from an old, now resolved, VPN bug that duplicated keys. After deleting those, Firefox 137 is working for me.

What VPN software caused the duplicate keys? I've been trying to pinpoint down why I had over 18,000 random private keys.

It was Forticlient, not sure exactly which versions had this bug, but that was the explanation I got from my IT department for the excessive keys.

Comment 19

[Tim Small]

(In reply to Gavin Heverly-Coulson from comment #18)

(In reply to Tim Small from comment #15)

(In reply to Gavin Heverly-Coulson from comment #14)

I can confirm that this has addressed my issue. I had over 67,000 private keys in my keychain, turned out 66,500 of them were from an old, now resolved, VPN bug that duplicated keys. After deleting those, Firefox 137 is working for me.

What VPN software caused the duplicate keys? I've been trying to pinpoint down why I had over 18,000 random private keys.

It was Forticlient, not sure exactly which versions had this bug, but that was the explanation I got from my IT department for the excessive keys.

That makes a lot of sense now and I kinda figured it was that. I work in the IT department where I work and we use FortiGate firewalls, so we use FortiClient for SSL VPN connections. I'm going to relay this to a Reddit thread that someone posted a week or so ago and see if the few people there have/had FortiClient on their devices at one point.

Comment 20

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1945985

Comment 21

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: Bug 1958140 - rsclientcerts: search for new objects at most once per search indicated by gecko r?jschanck

In bug 1945985, we attempted to make more explicit the mechanism by which gecko
indicates rsclientcerts that it should search for new objects. In that
implementation, if a low-priority background thread were searching for
certificates, other higher-priority threads (such as the networking thread)
could inadvertantly cause rsclientcerts to repeatedly and unnecessarily search
for objects. This could even completely stall networking if searching for
objects took more than a few seconds (say due to a bug in other software that
created tens of thousands of keychain items).

This patch improves on bug 1945985 by ensuring rsclientcerts can search at most
once every time gecko initiates a search.

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/34476d90db55
rsclientcerts: search for new objects at most once per search indicated by gecko r=jschanck

Comment 23

[Dana Keeler (she/her) (use needinfo) [:keeler]]

Attachment: Bug 1958140 - rsclientcerts: search for new objects at most once per search indicated by gecko r?jschanck

In bug 1945985, we attempted to make more explicit the mechanism by which gecko
indicates rsclientcerts that it should search for new objects. In that
implementation, if a low-priority background thread were searching for
certificates, other higher-priority threads (such as the networking thread)
could inadvertantly cause rsclientcerts to repeatedly and unnecessarily search
for objects. This could even completely stall networking if searching for
objects took more than a few seconds (say due to a bug in other software that
created tens of thousands of keychain items).

This patch improves on bug 1945985 by ensuring rsclientcerts can search at most
once every time gecko initiates a search.

Original Revision: https://phabricator.services.mozilla.com/D245794

Comment 24

[Phabricator Automation]

beta Uplift Approval Request

• User impact if declined: for affected users (with unexpectedly large keychains), Firefox won't load any URLs
• Code covered by automated testing: yes
• Fix verified in Nightly: no
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: low
• Explanation of risk level: this is a small change
• String changes made/needed: none
• Is Android affected?: no

Comment 25

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/34476d90db55

Comment 26

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-beta/rev/de05aa9342d1