Open Bug 1694035 Opened 4 years ago Updated 3 months ago

HTTPS request freezes when visiting google related services (Google, YouTube, ...)

Categories

(Core :: Networking: HTTP, defect, P2)

Product:
Core
Component:
Networking: HTTP
Version:
Firefox 85
Type:
defect

Tracking

()

Status:
UNCONFIRMED

People

(Reporter: bugzilla.mozilla, Unassigned)

References

Details

(Whiteboard: [necko-triaged])

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0

Steps to reproduce:

  • Visit Google related services (Youtube, Google search, Google Account Login) a few times.
  • After some normal usage (a few Minutes with a few requests) the described behavior occurs. When opening a tab some time later it works as intended for a small duration.
    This can be reproduced on multiple of my devices running an up to date version of Firefox on Arch Linux as well as Windows 10. Enabling and disabling features like the tracking protection or DNS over HTTPS seem to have no effect. All page data and cookies where cleared. The same is true for the private mode. The described websites are accessible as expected using Google Chrome, Chromium and wget.

Actual results:

When using a google related service such as YouTube, it works as intended for a few Minutes, but after some time requests stop terminating and stay open for ever (I closed the browser after a few hours). Reloading the tab after a few Minutes helps but that is also only temporary.
A profiling of Firefox (https://share.firefox.dev/3pzVSqg) shows, that the requests seem to hang before they are send out to Google. The network tab in the developer mode shows no requests at all in this case.

Expected results:

The request should be send to Google and return a response. The request should also timeout if that is not possible.

The Bugbug bot thinks this bug should belong to the 'Core::Networking: DNS' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Component: Untriaged → Networking: DNS
Product: Firefox → Core

Thank you for the bug report.

It's difficult to figure out the cause given the profile, so if possible we'd need you to answer a few questions:

  1. Does this happen with a fresh profile (without extensions)? https://support.mozilla.org/en-US/kb/profile-manager-create-remove-switch-firefox-profiles#w_creating-a-profile
  2. Does this happen with the latest nightly? https://nightly.mozilla.org/
  3. Are you using a proxy?
  4. Did this start happening shortly after an update?
  5. Could you capture some HTTP logs for us? (preferably using the new profile, as it could contain private data). https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

Thanks!

Flags: needinfo?(bugzilla.mozilla)

(In reply to Valentin Gosu [:valentin] (he/him) from comment #2)

Thank you for the bug report.

It's difficult to figure out the cause given the profile, so if possible we'd need you to answer a few questions:

  1. Does this happen with a fresh profile (without extensions)? https://support.mozilla.org/en-US/kb/profile-manager-create-remove-switch-firefox-profiles#w_creating-a-profile
  2. Does this happen with the latest nightly? https://nightly.mozilla.org/
  3. Are you using a proxy?
  4. Did this start happening shortly after an update?
  5. Could you capture some HTTP logs for us? (preferably using the new profile, as it could contain private data). https://developer.mozilla.org/en-US/docs/Mozilla/Debugging/HTTP_logging

Thanks!

Hi Valentin,

thank you for your fast reply.

  • I installed the nightly build with an fresh profile. The issue still occurred.
  • I am also not using a proxy, vpn, adblocker or anything like this.
  • The issue is present for a few months / a couple of weeks now. Sadly I did not noted the date when it first hapend. I update my system on a daily bases so I am not able to answer this point.
  • I uploaded my logs here: https://cloud.mafiasi.de/s/i6KEZHgMzw6iXTb
    The log ends with the issue occurring (~18:32). I was on the google search page for the term "opera" (I googled some browser names to reproduce the issue). When I clicked on the link to the Wikipedia page of opera it froze. As all links on this page it first pointed to the forwarding/analytics page hosted by google.

I hope this information helps!

Cheers

Flags: needinfo?(bugzilla.mozilla)

Valentin, would you have the time to check the provided HTTP logs? Maybe that is similar to what I see (bug 1648101) given that I also hit these kind of hangs for Google related domains (eg. especially when sites want to load fonts).

Flags: needinfo?(valentin.gosu)

Thank you for the logs.
This seems to be an issue with the http2 code.

What I see is that everything is fine until one request fails with code (804B0014 = NS_ERROR_NET_RESET). The following requests all fail with the same code.

2021-02-23 18:32:36.408111 UTC - [Parent 71316: Socket Thread]: V/nsHttp nsHttpConnection::CloseTransaction[this=0x7fa60bc09400 trans=0x7fa60e477800 reason=80004004]
2021-02-23 18:32:36.408115 UTC - [Parent 71316: Socket Thread]: V/nsHttp nsHttpConnection::DontReuse 0x7fa60bc09400 spdysession=0x7fa60e477800
2021-02-23 18:32:36.408118 UTC - [Parent 71316: Socket Thread]: I/nsHttp Http2Session::DontReuse 0x7fa60e477800
2021-02-23 18:32:36.408122 UTC - [Parent 71316: Socket Thread]: V/nsHttp   closing associated mTransaction
2021-02-23 18:32:36.408125 UTC - [Parent 71316: Socket Thread]: I/nsHttp Http2Session::Close 0x7fa60e477800 80004004
2021-02-23 18:32:36.408130 UTC - [Parent 71316: Socket Thread]: I/nsHttp Http2Session::CloseStream 0x7fa60e477800 0x7fa609cec080 0x139 804B0014
2021-02-23 18:32:36.408134 UTC - [Parent 71316: Socket Thread]: I/nsHttp MaybeDecrementConcurrent 0x7fa60e477800 id=0x139 concurrent=8 active=1
2021-02-23 18:32:36.408139 UTC - [Parent 71316: Socket Thread]: D/nsHttp nsHttpTransaction::Close [this=0x7fa60a93e000 reason=804b0014]

The NS_ERROR_NET_RESET from Http2Session::CloseStream seems to be coming from here triggered by this

What I can't see from the logs is if this error keeps affecting the following requests. @Dragana, could you take a look if this is indicative of a bug?

Component: Networking: DNS → Networking: HTTP
Flags: needinfo?(valentin.gosu) → needinfo?(dd.mozilla)

Http2Session::CloseStream is triggered by a shutdown.
the problem here is that server is not responding. There is a long lived HTTP/2 connection, but a some point the server stops responding.
TCP connection is still alive, at lease the OS thinks that TCP connection is still alive.

This is a server issue.

Flags: needinfo?(dd.mozilla)

The only think I may think of, is that something change around the time I was refactoring nsHttpConnectionMgr and that accidentally we keep connection open for a longer time. Although, it is very unlikely.

The change landed in Firefox 84.
May I ask you to download Firefox 83 and try to reproduce he issue?
You can find i here (for linux):
https://ftp.mozilla.org/pub/firefox/releases/83.0/linux-x86_64/en-US/

Flags: needinfo?(bugzilla.mozilla)

Florian, if you would have a bit of time to check if that problem is existent in Firefox 83 or not that would be kinda appreciated. Thanks!

Sorry for late answer. I tested Version 83 on multiple systems and the issue still occurs. I now try some binary search to localize the broken version.

Flags: needinfo?(bugzilla.mozilla)

I now removed an old T-Com Speedport W724V which was used as an access point and switch from my network. It was the only common failure point between the devices. After some testing, this issue seems to be gone. It's still strange, that this issue only affected google pages in firefox and not in other browsers. Maybe the chromium-based ones are more resistant to these kinds of timeouts and the router didn't support some fancy stuff google did with its requests? Not my field of research, but that's just my guess.
Sorry for the inconvenience. I only tested all of this with my internet connection, because of CoVid and the fact that the workstation isn't very portable.

Thanks a lot for all the details Florian! Dragana or Valentin, do you have an idea what this could be?

Flags: needinfo?(valentin.gosu)
Flags: needinfo?(dd.mozilla)

I added the Speedport again and upgraded its firmware. The issue seems to be gone.

Regarding the binary search which I originally intended. It happened with the old configuration to all tested Firefox versions. I went as low as Firefox 70.

he only difference between Chrome and Firefox may be that they are more aggressive with closing ideal HTTP/2 connection. We may try to tests when is Chrome closing idle connections.

Flags: needinfo?(dd.mozilla)
Priority: -- → P2
Whiteboard: [necko-triaged]

My keyboards sometimes does not type 't' :(
"The only difference .....

Flags: needinfo?(valentin.gosu)
Flags: needinfo?(valentin.gosu)

I think I have the same problem.
When connected through WIFI no problem.
Wired connection through a Netgear GS105, requests to google are freezing
Wired connection without Netgear hub no problem.
With chrome or edge no problem.
Weird :-)

Sorry thought it was the netgear stuff but just after I submitted my previous comment it happened again (this time without netgear hub). My wired connection goes through our enterprise network and I have no information on devices that are used. Weirder

I think the speedport was not the issue for me. I deactivated IPv6 for my System and this seemed to solve this issue. Not the best solution, but it fixed it in some way.

(In reply to Florian Vahl from comment #17)

I think the speedport was not the issue for me. I deactivated IPv6 for my System and this seemed to solve this issue. Not the best solution, but it fixed it in some way.

Disabling IPv6 has fixed my issue as well. Youtube and Reddit were having similar issues (embedded images were not loading, HTTP requests were not being responded to).

Hi, I'm encountering temporary (but frequent) problems during google search, too. This bug might fit the description.

I tried most of the recommendations I found online and can still reproduce it using multiple different devices. It's not a mobile or desktop issue. I suspect the service serving the intermidiate page might be the culprit: https://www.ghacks.net/2021/09/30/dont-track-me-google-for-firefox-blocks-links-manipulations-by-google-when-clicking-or-copying/?amp

It makes Firefox+Google unusable for me, currently I am testing the addon. It works nicely so far. However, if this suspicion proves correct, it might lead to lots of users lost to Chrome. Even if it's just a temporary load problem on Google's side, it is there since weeks and might evolve into a bigger issue. So I hope that I'm wrong, and it's just my ISP or local setup, but urge you to investigate.

Thanks and keep up the great work,

David

Judging by what others have posted, it's pretty clear there's a correlation with IPv6 usage.
I wasn't able to find any good leads in the logs.

If anyone can still reproduce this, I'd appreciate some more logs using Firefox Nightly.
Thanks!

Flags: needinfo?(valentin.gosu)
Severity: -- → S3

Started experiencing this same issue after Verizon enabled IPv6 in my area, but after some digging around I found that this is most likely a bug between your ISP equipment and networking setup. Specifically it has to do with Intel NICs dropping packets due to bad checksum data being inserted somewhere upstream.

Here's a Reddit thread on this issue: https://www.reddit.com/r/Fios/comments/v1ttcl/router_firmware_update_31117_causing_massive/

If you have fiber internet to your home that terminates at an ONT, you are on a wired connection, and your network adapter uses an Intel NIC, try disabling TCP checksum offload for IPv6 in your adapter settings: https://www.intel.com/content/www/us/en/download/19174/disabling-tcp-ipv6-checksum-offload-capability-with-intel-1-10-gbe-controllers.html This cleared up the issue right away for me.

You need to log in before you can comment on or make changes to this bug.