1979124 - Network connections seem to stall and application must be force quit

Status: RESOLVED FIXED

(Core :: Networking, defect, P1)

Description

[Geoff Hing]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:141.0) Gecko/20100101 Firefox/141.0

Steps to reproduce:

• On macOS Sequoia 15.5, create an additional profile using about:profiles.
• Close Firefox.
• Reopen Firefox and use about:profiles to select the newly-created profile to open in a new window.

This behavior also appeared when using the new profile system (https://support.mozilla.org/en-US/kb/profile-management#w_create-a-new-profile|Manage).

I first started experiencing this issue after the update to 140.0.

Actual results:

• Some of the suggested story card images fail to load.
• After entering an address in the location bar, the page does not load.
• Sometimes the rainbow spinner icon will appear over the window.
• In order to close the application, I have to force quit the application using the contextual menu for the dock icon.

Expected results:

• The page with the URL entered in the address bar should be loaded.

Comment 1

[BugBot [:suhaib / :marco]]

The Bugbug bot thinks this bug should belong to the 'Toolkit::Startup and Profile System' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Geoff Hing]

I originally experienced this bug when switching between profiles, but upon further testing, I have noticed that this bug occurs even when using the default profile and without creating explicit profiles. I have updated the subject to reflect this and here is updated reproduction information to reflect seeing this issue even when not using multiple profiles.

Steps to reproduce:

• Start with a clean install of Firefox 141.0.
• Restart the computer.
• Open Firefox.

Actual results:

• Some of the suggested story card images fail to load.
• After entering an address in the location bar, the page does not load.
• Sometimes the rainbow spinner icon will appear over the window.
• In order to close the application, I have to force quit the application using the contextual menu for the dock icon.
• The crash reporter dialog is displayed.

Expected results:

• The page with the URL entered in the address bar should be loaded.

This issue continues to be present even after upgrading to 141.0.2.

Comment 3

[Geoff Hing]

Also, I'm seeing this issue on a computer with an Apple M1 chipset. I wasn't sure of the Hardware field value that corresponds to that.

Comment 4

[Markus Stange [:mstange]]

Could I ask you to collect a profile of the Firefox process by sending signals from the terminal? See https://profiler.firefox.com/docs/#/./async-posix-signal-control for details. You can upload the collected JSON file on https://profiler.firefox.com/ and then share the link here. Record maybe 20 seconds of you trying to load new pages and interacting with the UI while Firefox is in this state. Thank you!

Comment 5

[Geoff Hing]

@mstrange, I tried to capture the profile using the instructions that involved sending the signals to the Firefox process that you shared. However, after running those commands, no JSON files were created in my ~/Downloads folder.

I am able to start recording a profile through devtools, as described at https://profiler.firefox.com/docs/#/./guide-getting-started, however, clicking the "Capture recording" button fails because of the very issue I've reported here.

Is there some other place the profile data might be stored, or another way to start the profiler?

Comment 6

[Markus Stange [:mstange]]

There is another way to capture profiles if the UI is working but pages aren't loading:

1. Start the profiler through the UI.
2. Record the interaction.
3. Capture the profile via the Browser Console (Cmd+Shift+J), see below.
4. Stop the profiler through the UI.

This is the command to run on the browser console (replace it with the correct path):

await Services.profiler.dumpProfileToFileAsync("/Users/yourusername/Downloads/profile.json");

Comment 7

[Geoff Hing]

I was able to capture the profile using that method. Here is the profile URL: https://profiler.firefox.com/from-file/calltree/?globalTrackOrder=de0c61732a8954b&hiddenGlobalTracks=1wc&hiddenLocalTracksByPid=32297-23~32306-0~32307-0~32305-0~32304-0~32301-0~32521-0~32353-0~32461-0~32356-0~32303-0~32299-0~32300-0w2&thread=0&v=11.

Comment 8

[Markus Stange [:mstange]]

Sorry, the link doesn't work - you also need to upload the profile using the button + panel in the top right corner. Could you upload it please? Thanks!

Comment 9

[Geoff Hing]

Ok, I think this link should do it: https://share.firefox.dev/3J2wjxy

Comment 10

[Markus Stange [:mstange]]

Yes, that worked! Unfortunately I have to ask for another profile. Could you go to about:logging, pick the Networking preset, click "Set Log Modules", and click "Start Logging". Then record similar actions as in the current profile, and then capture it the same way as you did last time. Thanks!

Comment 11

[Geoff Hing]

Here is a profile created after using the Networking preset logging modules as instructed in your last message: https://share.firefox.dev/46NzH9s

Comment 12

[Markus Stange [:mstange]]

Thanks! The profile shows that the socket thread is stuck in an NSS Lock: https://share.firefox.dev/40RvHB2
I don't know which other thread is currently holding the lock. Kershaw, can you take a look?

Comment 13

[Kershaw Chang [:kershaw]]

(In reply to Markus Stange [:mstange] from comment #12)

Thanks! The profile shows that the socket thread is stuck in an NSS Lock: https://share.firefox.dev/40RvHB2
I don't know which other thread is currently holding the lock. Kershaw, can you take a look?

Based on the profile in comment #11, there’s no activity on the socket thread. It's clear that the socket thread is blocked.
Looking at the stack trace, the blocking appears to be caused by a lock on nssSession inside NSSToken here. Unfortunately, I’m not very familiar with NSS internals, so I’m not sure why another thread is able to acquire that lock before the socket thread.

I think the real question is: why do we allow other threads to take this lock ahead of the socket thread?

Maybe John has some insight?

Comment 14

[John Schanck [:jschanck]]

This call to certdb.asyncHasThirdPartyRoots() would take the same lock on a background thread. Hard to say for sure that this is the issue. But to narrow it down, maybe there is a way to prevent the DoH heuristics from running in a new profile?

Comment 15

[Valentin Gosu [:valentin]]

(In reply to John Schanck [:jschanck] (out of office Aug 11-18) from comment #14)

This call to certdb.asyncHasThirdPartyRoots() would take the same lock on a background thread. Hard to say for sure that this is the issue. But to narrow it down, maybe there is a way to prevent the DoH heuristics from running in a new profile?

DoHHeuristics can be disabled by setting doh-rollout.disable-heuristics to true, or setting network.trr.mode to 5.

Comment 16

[Geoff Hing]

Disabling DoHHeuristics by setting network.trr.mode to 5 in the settings accessible by about:config, then restarting Firefox, seems to allow network traffic. That is, after changing that setting, I am able to browse normally and do not experience the behavior I reported originally in this issue.

Comment 17

[Valentin Gosu [:valentin]]

Thanks you for the confirmation.
@John, do you already know if there's something wrong with the implementation of nsNSSCertificateDB::AsyncHasThirdPartyRoots
I don't understand why calling this on a background thread would hold the lock forever.

Comment 18

[Markus Stange [:mstange]]

Geoff, would you mind reproducing the hang once more, and then getting another profile where you check "Bypass selections above and record all registered threads" on about:profiling? It might let us see which other thread is holding the lock at that point.

Comment 19

[Valentin Gosu [:valentin]]

(In reply to Valentin Gosu [:valentin] (he/him) {{ AFK in July }} from comment #17)

Thanks you for the confirmation.
@John, do you already know if there's something wrong with the implementation of nsNSSCertificateDB::AsyncHasThirdPartyRoots
I don't understand why calling this on a background thread would hold the lock forever.

Redirecting to Dana as John is out of office.

Comment 20

[Geoff Hing]

(In reply to Markus Stange [:mstange] from comment #18)

Geoff, would you mind reproducing the hang once more, and then getting another profile where you check "Bypass selections above and record all registered threads" on about:profiling? It might let us see which other thread is holding the lock at that point.

Markus, what should I set network.trr.mode to to create the requested profile since I changed that setting to check whether that bypassed the issue I was having?

Comment 21

[Markus Stange [:mstange]]

If that's what's needed to reproduce the issue, yes, I'm afraid so. Maybe you could try it in Firefox Nightly or in a separate profile so that it doesn't disrupt your regular use of the browser too much?

Comment 22

[Dana Keeler (she/her) [:keeler]]

(In reply to Valentin Gosu [:valentin] (he/him) from comment #17)

Thanks you for the confirmation.
@John, do you already know if there's something wrong with the implementation of nsNSSCertificateDB::AsyncHasThirdPartyRoots
I don't understand why calling this on a background thread would hold the lock forever.

We know NSS has some fundamental issues with threads and locking, but we're not aware of a specific issue with the code that function uses. That function is only used by the doh heuristics, though, so we can sidestep the issue by removing it, as outlined in bug 1959526.

Comment 23

[Geoff Hing]

(In reply to Markus Stange [:mstange] from comment #18)

Geoff, would you mind reproducing the hang once more, and then getting another profile where you check "Bypass selections above and record all registered threads" on about:profiling? It might let us see which other thread is holding the lock at that point.

Markus, here's the profile I took after I checked "Bypass selections above and record all registered threads" on about:profiling:

https://share.firefox.dev/41Dg00o

Comment 24

[Markus Stange [:mstange]]

Perfect!

Dana, the NSS session lock is being held by a BgIOThreadPool thread during a find_objects callback from NSS. That thread is trying to run the find_objects work synchronously on the osclientcert thread, but the osclientcert thread is also trying to synchronously drop a ThreadSpecificHandles on the identity key's thread. I'm not sure which thread that is - maybe the osclientcert thread is deadlocking with itself? Or it might be deadlocking with one of the other two threads.

Comment 25

[Markus Stange [:mstange]]

Profile: https://share.firefox.dev/4oy5NfL

Comment 26

[BugBot [:suhaib / :marco]]

Set release status flags based on info from the regressing bug 1962280

Comment 27

[Markus Stange [:mstange]]

I don't have much evidence that this is actually a regression from bug 1962280 - that bug was just the last bug that the code which is implicated in the deadlock was touched.

Comment 28

[Markus Stange [:mstange]]

Oh, maybe this bug was already fixed in bug 1977347?

Comment 30

[Kershaw Chang [:kershaw]]

After discussion, we've decided to remove nsNSSCertificateDB::AsyncHasThirdPartyRoots.

Comment 31

[Kershaw Chang [:kershaw]]

Attachment: Bug 1979124 - Remove thirdPartyRoots check in DoHHeuristics, r=#necko,keeler

Comment 33

[Markus Stange [:mstange]]

Can we do anything about Firefox 141 in the remaining 6 days before Firefox 142 gets released with the fix from bug 1977347? Is there a pref we can turn off remotely?

Comment 34

[Kershaw Chang [:kershaw]]

(In reply to Markus Stange [:mstange] from comment #33)

Can we do anything about Firefox 141 in the remaining 6 days before Firefox 142 gets released with the fix from bug 1977347? Is there a pref we can turn off remotely?

I guess we could turn off DoH via remote settings, but I don’t think we want to do that.
Dana, do you have any idea how many users might be affected? Is it worth disabling DoH for all users because of this?

Comment 35

[Kershaw Chang [:kershaw]]

This can be downgraded to S2 (same as bug 1977347).

Comment 36

[Kershaw Chang [:kershaw]]

(In reply to Markus Stange [:mstange] from comment #33)

Can we do anything about Firefox 141 in the remaining 6 days before Firefox 142 gets released with the fix from bug 1977347? Is there a pref we can turn off remotely?

If I understand the root cause from bug 1977347 #comment 15 correctly, this likely won’t affect many users, so I think we can wait until the Firefox 142 release.

Comment 37

[Dana Keeler (she/her) [:keeler]]

(In reply to Kershaw Chang [:kershaw] from comment #34)

Dana, do you have any idea how many users might be affected? Is it worth disabling DoH for all users because of this?

(closing the loop here) Yeah, I don't think many users are affected - we would have heard of more instances of this issue by now.

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/098927d1e63f
https://hg.mozilla.org/integration/autoland/rev/d8c74940af78
Remove thirdPartyRoots check in DoHHeuristics, r=keeler,necko-reviewers,valentin

Comment 39

[Kershaw Chang [:kershaw]]

Attachment: Bug 1979124 - Remove thirdPartyRoots check in DoHHeuristics, r=#necko,keeler

Original Revision: https://phabricator.services.mozilla.com/D261008

Comment 40

[Phabricator Automation]

firefox-release Uplift Approval Request

• User impact if declined: Firefox could become unusable.
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: N/A
• Risk associated with taking this patch: Low
• Explanation of risk level: Removing the thirdPartyRoots check is low risk.
• String changes made/needed: N/A
• Is Android affected?: no

Comment 41

[Kershaw Chang [:kershaw]]

Not sure if it's too late to request uplift, but if we still have time, taking this low-risk patch should be fine.

Comment 42

[BugBot [:suhaib / :marco]]

The bug is marked as tracked for firefox142 (beta) and tracked for firefox143 (nightly). However, the bug still has low severity.

:ghess, could you please increase the severity for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 43

[Markus Stange [:mstange]]

(In reply to Dana Keeler (she/her) [:keeler] from comment #37)

(closing the loop here) Yeah, I don't think many users are affected - we would have heard of more instances of this issue by now.

We have bug 1977347, this bug, two or three duplicates of this bug, and we also have bug 1944078 and its duplicates which might be the same problem - at least some reports in that bug mention the UI being responsive but pages not loading, in 141. That's quite a high number of reports. And we probably don't have any insight from telemetry into these hangs because we can't submit telemetry without a working socket thread.

Comment 44

[Geoff Hing]

(In reply to Dana Keeler (she/her) [:keeler] from comment #37)

(In reply to Kershaw Chang [:kershaw] from comment #34)

Dana, do you have any idea how many users might be affected? Is it worth disabling DoH for all users because of this?

(closing the loop here) Yeah, I don't think many users are affected - we would have heard of more instances of this issue by now.

FWIW, as the original reporter, I have at least two colleagues who are affected by this bug, and have either switched to other browsers, or are just reinstalling Firefox and not restarting it, to try to avoid this bug.

Before submitting this bug report, I also found this similar report on Reddit.

Also, as this bug requires me to force quit Firefox, the crash reporter dialog typically appears, and I've been submitting this when I've had to force quit the browser. Is there crash report data that would help quantify the exposure to this bug? I don't know if the ability to submit those reports is also hindered by the socket issue, though it seems like the crash reporter is a separate process.

Also, albeit less frequently, I have submitted the Apple crash report through the dialog that appears when I have to force quit Firefox. I don't know if you all have access to insights from Apple's data, but that could also be a signal to exposure.

Comment 45

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/d8c74940af78

Comment 46

[Dana Keeler (she/her) [:keeler]]

(In reply to Markus Stange [:mstange] from comment #43)

(In reply to Dana Keeler (she/her) [:keeler] from comment #37)

(closing the loop here) Yeah, I don't think many users are affected - we would have heard of more instances of this issue by now.

We have bug 1977347, this bug, two or three duplicates of this bug, and we also have bug 1944078 and its duplicates which might be the same problem - at least some reports in that bug mention the UI being responsive but pages not loading, in 141. That's quite a high number of reports. And we probably don't have any insight from telemetry into these hangs because we can't submit telemetry without a working socket thread.

I'm not opposed to uplifting a fix or otherwise taking action to address this sooner if you think that's necessary. That said, 142 will be released shortly, which does have the fix.

Comment 47

[BugBot [:suhaib / :marco]]

The patch landed in nightly and beta is affected.
:kershaw, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
  • See https://wiki.mozilla.org/Release_Management/Requesting_an_Uplift for documentation on how to request an uplift.
• If no, please set status-firefox142 to wontfix.

For more information, please visit BugBot documentation.

Comment 48

[Ryan VanderMeulen [:RyanVM]]

The release approval request is good enough for 142 at this point, but it would be good to get this nominated for ESR140 also.

Comment 50

[Kershaw Chang [:kershaw]]

Attachment: Bug 1979124 - Remove thirdPartyRoots check in DoHHeuristics

Original Revision: https://phabricator.services.mozilla.com/D261008

Comment 51

[Ryan VanderMeulen [:RyanVM]]

Please fill out the uplift request form on the ESR140 patch.

Comment 52

[Phabricator Automation]

firefox-esr140 Uplift Approval Request

• User impact if declined: Firefox could become unusable becuse socket thread is blocked
• Code covered by automated testing: no
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: n/a
• Risk associated with taking this patch: Low
• Explanation of risk level: This patch only removes the code that could block socket thread
• String changes made/needed: n/a
• Is Android affected?: yes

Comment 53

[Kershaw Chang [:kershaw]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #51)

Please fill out the uplift request form on the ESR140 patch.

Sorry, I filled the form but forgot to submit.

Comment 54

[Pulsebot]

https://hg.mozilla.org/releases/mozilla-esr140/rev/a1257d8c043c