Closed Bug 1749910 (foxstuck) Opened 2 years ago Closed 2 years ago

[2022-01-13] Hangs in socket thread

Categories

(Core :: Networking, defect, P1)

Desktop
All
defect

Tracking

()

VERIFIED FIXED
Tracking Status
firefox-esr91 --- fixed
firefox96 blocking verified
firefox97 blocking verified
firefox98 blocking verified

People

(Reporter: mcoman, Unassigned)

References

Details

Crash Data

Attachments

(1 file)

[Affected versions]:

  • Firefox Release - Build ID: 20220106144528
  • Firefox Beta - Build ID: 20220111185943
  • Firefox Nightly - Build ID: 20220112213002

[Affected Platforms]:

  • Windows 10 x64 (Version 21H1 - 19043.1466)
  • Windows 10 x64 (Version 21H1 - 19043.1415)
  • Windows 10 x64 (Version 20H2 - 19042.1466)
  • macOS 11.6
  • Ubuntu 20.04

[Prerequisites]:

  • Have a previous version of Firefox Nightly installed.

[Steps to reproduce]:

  1. Open the browser from the prerequisites and update it.
  2. Navigate to any website (e.g. https://www.wikipedia.org/) and observe the behavior.

[Expected result]:

  • The website is successfully opened.

[Actual result]:

  • The website is not open.

[Notes]:

  • If a new update check is performed will be stuck in "Checking for updates..." and eventually will crash.
  • This issue is reproducible after a browser restart.
  • This issue is not reproducible with a new Firefox profile.
  • Attached a screen recording of the issue.

Could you try to get a http log?
Thanks.

Flags: needinfo?(mcoman)

Reported by several users and employees.

Did a remote settings update break anything?

On Windows, I get crash reports with @ shutdownhang | mozilla::net::nsHttpConnectionMgr::Shutdown on shudown.

For Pascal on Linux, Firefox instances are crashing after a while bp-c85ffd21-5681-4359-a7d6-73d440220113

Flags: needinfo?(mathieu)
See Also: → 1633342
See Also: → 1749908

https://www.reddit.com/r/firefox/comments/s2u7eg/is_firefox_down/

Windows 11 too

Disable HTTP3 via about:config is temp solution

about:networking logs are 100% of:

2022-01-13 08:52:05.261348 UTC - [Parent 440321: Socket Thread]: V/nsHttp nsHttpTransaction::ReadRequestSegment 7ff90e0b1000 read=0
2022-01-13 08:52:05.261350 UTC - [Parent 440321: Socket Thread]: V/nsHttp Http3Stream::OnReadSegment count=371 state=4 [this=7ff9059c3660]

Spamming at a very rapid rate.

This is not Windows only.

OS: Windows → All
Summary: [Windows] No websites can be opened after latest Firefox update → No websites can be opened after latest Firefox update

Hi Kershaw. The only thing I can find is the GET request from the "Network" tab of the "Web Developer Tools". I cannot save any log file from the about:networking page.

Also, it seems that the browser hangs when trying to close it and I must terminate the process from the "Task Manager" in order to restart it.

Flags: needinfo?(mcoman)

I mentioned in bug 1749908 that disabling the Firefox Data Collection makes Firefox work again.

Same here on macOS with Developer Edition (97.0b2 64-bit, build 20220111185943): bp-6006ac70-7dd3-44fa-a72e-fb99b0220113
I tried to record http log and it has created a +500Mb file in few seconds…

Summary: No websites can be opened after latest Firefox update → [2022-01-13] No websites can be opened after updating Firefox on any channel

(In reply to Nicolas from comment #8)

Same here on macOS with Developer Edition (97.0b2 64-bit, build 20220111185943): bp-6006ac70-7dd3-44fa-a72e-fb99b0220113
I tried to record http log and it has created a +500Mb file in few seconds…

Could you zip it and share? Could help to understand what triggers this hang.

Flags: needinfo?(nicolas.sandri)

(In reply to Kershaw Chang [:kershaw] from comment #9)

(In reply to Nicolas from comment #8)

Same here on macOS with Developer Edition (97.0b2 64-bit, build 20220111185943): bp-6006ac70-7dd3-44fa-a72e-fb99b0220113
I tried to record http log and it has created a +500Mb file in few seconds…

Could you zip it and share? Could help to understand what triggers this hang.

Sorry but I'm behind a DSL connection so it will take the day to upload…
I took a look at the content before deleting log, I hope this can help you:

head /private/var/folders/…/log.txt-main.558.moz_log

[Parent 558: Main Thread]: I/Logger Flushing old log files
2022-01-13 08:54:49.195771 UTC - [Parent 558: Socket Thread]: V/nsHttp Http3Stream::OnReadSegment count=337 state=4 [this=14c7bc500]
2022-01-13 08:54:49.195856 UTC - [Parent 558: Socket Thread]: V/nsHttp nsHttpTransaction::ReadRequestSegment 1433f2000 read=0
2022-01-13 08:54:49.195874 UTC - [Parent 558: Socket Thread]: V/nsHttp Http3Stream::OnReadSegment count=337 state=4 [this=14c7bc500]
2022-01-13 08:54:49.195919 UTC - [Parent 558: Socket Thread]: V/nsHttp nsHttpTransaction::ReadRequestSegment 1433f2000 read=0
2022-01-13 08:54:49.195963 UTC - [Parent 558: Socket Thread]: V/nsHttp Http3Stream::OnReadSegment count=337 state=4 [this=14c7bc500]
2022-01-13 08:54:49.195988 UTC - [Parent 558: Socket Thread]: V/nsHttp nsHttpTransaction::ReadRequestSegment 1433f2000 read=0
2022-01-13 08:54:49.196039 UTC - [Parent 558: Socket Thread]: V/nsHttp Http3Stream::OnReadSegment count=337 state=4 [this=14c7bc500]
2022-01-13 08:54:49.196093 UTC - [Parent 558: Socket Thread]: V/nsHttp nsHttpTransaction::ReadRequestSegment 1433f2000 read=0
2022-01-13 08:54:49.196114 UTC - [Parent 558: Socket Thread]: V/nsHttp Http3Stream::OnReadSegment count=337 state=4 [this=14c7bc500]

Flags: needinfo?(nicolas.sandri)

@Kershaw, I have managed to get the log file (1.88 GB). You can find it here.

Can reproduce. Note that this also caused shutdown hang (stuck at "profile-change-net-teardown") for me - searched this keyword, couldn't find any tickets, and finally reached this ticket after looking at the top crasher in CrashStats.

I'm using this bug to duplicate all the incoming bugs, and avoid further noise in the bug where the investigation is likely going to happen (bug 1749908).

For folks affected, you can try opening about:config and set network.http.http3.enabled to false as a temporary workaround.

Not sure if the ec2-44-233-180-72.us-west-2.compute.amazonaws.com URL (why is even an Amazon EC2 instance used?) from https://bugzilla.mozilla.org/show_bug.cgi?id=1749924 is relevant (because it never closes the connection as it seems). At least nothing has changed in the browser itself. Or was an update pushed by Mozilla, that we are not aware of?

Or was an update pushed by Mozilla, that we are not aware of?

It affects all Firefox versions, even ESR, so the suspicion is that an external service updated (at exactly 12AM PST), specifically one that may load balance for our telemetry/update/blocklist servers.

Flags: needinfo?(mathieu)
Summary: [2022-01-13] No websites can be opened after updating Firefox on any channel → [2022-01-13] Hangs in socket thread

Firefox has witnessed outages and we are sorry for that. We believe it's fixed and a restart of Firefox should restore normal behaviour. We will provide more information shortly

Status: NEW → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED

Did this issue result in an increase in CPU usage for anyone while Firefox was unable to establish online connectivity? If so then I think I was also hit by the bug, it hasn't reoccurred since restarting Firefox.

I have verified that this issue is no longer reproducible with the latest Firefox Release (96.0 Build ID - 20220106144528), Firefox Beta (97.0b2 Build ID - 20220111185943), and Firefox Nightly (98.0a1 Build ID - 20220112213002) installed on Windows 10 x64, macOS 11.6, and Linux Mint 20.2 x64. Now I can confirm that any website can be opened after a browser restart, however, if the affected profile is still opened we need to terminate the process since the browser hangs when closing.

Depends on: 1749957
Alias: foxstuck
Crash Signature: [@ shutdownhang | __psynch_cvwait | _pthread_cond_wait | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | __pthread_cond_wait | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | mozilla::net::nsHttpConnectionMgr::Shutdown] [@…
Crash Signature: [@ shutdownhang | __psynch_cvwait | _pthread_cond_wait | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | __pthread_cond_wait | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | mozilla::net::nsHttpConnectionMgr::Shutdown] [@… → [@ shutdownhang | __psynch_cvwait | _pthread_cond_wait | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | __pthread_cond_wait | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | __psynch_cvwait | mozilla::net::nsHttpConnection…
Crash Signature: ntdll.dll | _tailMerge_d3dcompiler_47.dll | kernelbase.dll | xul.dll | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | ntdll.dll | kernelbase.dll | mozilla::net::nsHttpConnectionMgr::Shutdown] → ntdll.dll | _tailMerge_d3dcompiler_47.dll | kernelbase.dll | xul.dll | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | ntdll.dll | kernelbase.dll | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ shutdownhang | __pthread_cond_wait | mozi…
Crash Signature: mozilla::TaskController::GetRunnableForMTTask | mozilla::net::nsHttpConnectionMgr::Shutdown] → mozilla::TaskController::GetRunnableForMTTask | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ EMPTY: no crashing thread identified; EmptyMinidump]

shutdownhang | mozilla::net::nsHttpConnectionMgr::Shutdown is still a top crash for Firefox esr 102
https://crash-stats.mozilla.org/topcrashers/?product=Firefox&version=102.6.0esr

Crash Signature: mozilla::TaskController::GetRunnableForMTTask | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ EMPTY: no crashing thread identified; EmptyMinidump] → mozilla::TaskController::GetRunnableForMTTask | mozilla::net::nsHttpConnectionMgr::Shutdown] [@ EMPTY: no crashing thread identified; EmptyMinidump]
You need to log in before you can comment on or make changes to this bug.