Closed Bug 1866925 Opened 2 years ago Closed 2 years ago

network.auth.use_redirect_for_retries sometimes causes reading responses to never complete on Windows

Categories

(Core :: Networking, defect, P1)

Product:
Core
Component:
Networking
Version:
Firefox 121
Type:
defect

Tracking

()

Status:
RESOLVED DUPLICATE of bug 1871533

People

(Reporter: me+bugzilla, Assigned: smayya)

References

Details

(Whiteboard: [necko-triaged][necko-priority-next])

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0

Steps to reproduce:

This bug is tricky to reproduce. I have not yet had time to successfully reproduce this issue with servers under my control, so please bear with me.

On one Windows computer, the most reliable way to cause the issue to happen is to log into fidelity.com, browse around for a bit (pages will slowly start breaking), logging out, then trying to log back in. Some requests will hang, and the result is that logging back in is impossible depending on which requests hang. Other websites sometimes exhibit this problem too.

On another Windows computer that is behind a Bluecoat corporate proxy, I am experiencing the issue much more acutely with nearly every single website (microsoft.com seems to reliably trigger the issue). Disabling HTTP/3, HTTP/2, and TLS 1.3 does not fix the problem, nor does changing the user agent to match an unaffected version. I was able to reproduce the issue on a clean profile on version 121.0b4 and 121.0b1, but not 120 (or 115 ESR). At one point on 121.0b4, I tried downloading the installer for a previous version from ftp.mozilla.org, and it appeared to completely download the file but never actually complete the download.

Yes, I tried rebooting all the things, and yes I also checked that MTU was configured correctly.

Actual results:

See above, Firefox appears to hang for a long time after reading response bodies, causing various issues with websites. Some requests that weren't affected will be "blocked" for an extended period of time while I presume they wait for an available connection.

Expected results:

Firefox does not hang after reading response bodies.

Since the issue seems to be triggered almost immediately on the computer behind the corporate proxy, I suspect the issue is related to multiple requests/connections/domains(?) to the same IP address (about:networking -> sockets shows only the corporate proxy's IP on the second computer). I also notice that after triggering the issue, other websites that also use the same CDN provider (usually Akamai or Cloudfront or similar) also sometimes start having problems.

I have not been experiencing this problem on macOS.

Bug #1865968 appears to be an incarnation of this problem.

Clarification: when I said Firefox hangs, the browser itself does not hang (it remains responsive). The specific request(s) affected by the issue never complete, causing the page load (or whatever else is depending on the request completing) to also never complete.

The Bugbug bot thinks this bug should belong to the 'Core::Networking' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Component: Untriaged → Networking
Product: Firefox → Core

Hi Andrew, could you capture some HTTP logs for us?
https://firefox-source-docs.mozilla.org/networking/http/logging.html

If you say that you could reproduce this, starting with 121, could you also try using mozregression to determine what introduced the bug?
https://mozilla.github.io/mozregression/quickstart.html - let me know if you need help with this tool.

Thanks!

Flags: needinfo?(me+bugzilla)

I thought I had produced a simple test case to reproduce this problem, albeit in a very artificial way:

  1. Open a webpage (I used https://andrewsun.com/fx121/test2.html, results may vary depending on what the webserver supports)
  2. Set network.http.http2.enabled (or network.http.spdy.enabled) to false in about:config
  3. Reload the webpage (subsequent requests to that origin will take a very long time)

Profiler link: https://share.firefox.dev/3R0wjP1

Unfortunately, this actually seems to reproduce a similar but different bug. I bisected all the way back to between 2017-04-13 and 2017-04-14, after which it appears no builds were retained.

I am out of time for now, if I have more time later I will continue to try to reproduce this specific bug in a consistent way.

(In reply to Andrew Sun from comment #4)

I thought I had produced a simple test case to reproduce this problem, albeit in a very artificial way:

  1. Open a webpage (I used https://andrewsun.com/fx121/test2.html, results may vary depending on what the webserver supports)
  2. Set network.http.http2.enabled (or network.http.spdy.enabled) to false in about:config
  3. Reload the webpage (subsequent requests to that origin will take a very long time)

Profiler link: https://share.firefox.dev/3R0wjP1

Unfortunately, this actually seems to reproduce a similar but different bug. I bisected all the way back to between 2017-04-13 and 2017-04-14, after which it appears no builds were retained.

I am out of time for now, if I have more time later I will continue to try to reproduce this specific bug in a consistent way.

Thanks for the test case, but I think this is a different issue than what you described in comment #0. That's because there is no proxy used.

Could you try to capture a http log if you can reproduce the case in comment #0? It'd be great if you can select Logging to a file in about:logging, since a log file is easier to analyze.
Thanks.

I think I was able to reproduce the issue while logging with the following:

  1. start mitmproxy in non-intercepting mode with mitmweb --allow-hosts 127\.0\.0\.1:443 or similar (just as a convenient HTTP proxy)
  2. configure Firefox to use the proxy for HTTP and HTTPS
  3. open microsoft.com and browse around until things break

It seems to help to open the developer tools and check the "Disable Cache" box and do things like close the tabs and open microsoft.com again.

Here is the log that I produced from this browsing session: https://asun.co/s/log.txt-main.36164.moz_log.gz

I am trying to figure out a faster and more reliable way to reproduce this so bisecting is easier.

(In reply to Kershaw Chang [:kershaw] from comment #5)

(In reply to Andrew Sun from comment #4)
Thanks for the test case, but I think this is a different issue than what you described in comment #0. That's because there is no proxy used.

I should probably note that no proxy is needed to cause this issue (see the first example with fidelity.com) but it seems to cause the problem to happen much more easily.

I bisected the nightly builds on the work laptop where reproducing the problem is extremely easy, and found that things broke between 2023-10-24-09-38-27 and 2023-10-24-21-57-36. I did not bisect the autoland builds, but looking through the changes pushed between those two builds led me to https://hg.mozilla.org/mozilla-central/rev/dc3642b7cf56 - Bug 1856288 - Enable pref network.auth.use_redirect_for_retries for early beta.

Setting network.auth.use_redirect_for_retries to false fixes the problem on the work laptop. It makes sense this option is related to the problem since the Bluecoat proxy uses Kerberos authentication.

In my brief testing, setting the option to false on the other Windows computer also appears to solve the weirdness I have observed. I don't understand why this option would affect accessing things that don't seem to have HTTP authentication.

Flags: needinfo?(me+bugzilla) → needinfo?(smayya)
Summary: Firefox 121 beta hangs on reading the end of HTTP responses on Windows → network.auth.use_redirect_for_retries sometimes causes reading responses to never complete on Windows

Thank you for bisecting this, Andrew.
The pref is guarded by IS_EARLY_BETA_OR_EARLIER, so it's good to know it doesn't affect Firefox release.
We'll address this as soon as possible. Once it's fixed we'll needinfo you to check that you're no longer encountering any issues.

Severity: -- → S3
Priority: -- → P1
Whiteboard: [necko-triaged][necko-priority-queue]

I should probably add that what I did to reproduce the problem when bisecting is navigating to https://docs.oracle.com/en/java/javase/17/docs/api/index.html (arbitrarily-chosen test subject with static content) and doing a shift-refresh while behind a Kerberos-authenticated HTTP proxy (Bluecoat). The reload will never finish.

I don't have time at the moment to test whether it happens with a proxy using HTTP basic auth or other scheme, nor have I figured out what it is about certain websites (that don't seem to use HTTP authentication) that causes connections to seize up even without a proxy.

Assignee: nobody → smayya
Flags: needinfo?(smayya)
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

(In reply to Andrew Sun from comment #8)

I bisected the nightly builds on the work laptop where reproducing the problem is extremely easy, and found that things broke between 2023-10-24-09-38-27 and 2023-10-24-21-57-36. I did not bisect the autoland builds, but looking through the changes pushed between those two builds led me to https://hg.mozilla.org/mozilla-central/rev/dc3642b7cf56 - Bug 1856288 - Enable pref network.auth.use_redirect_for_retries for early beta.

Setting network.auth.use_redirect_for_retries to false fixes the problem on the work laptop. It makes sense this option is related to the problem since the Bluecoat proxy uses Kerberos authentication.

In my brief testing, setting the option to false on the other Windows computer also appears to solve the weirdness I have observed. I don't understand why this option would affect accessing things that don't seem to have HTTP authentication.

Hello Andrew,
Thanks for the logs.

We checked the logs from this https://asun.co/s/log.txt-main.36164.moz_log.gz and could not find anything related to authentication and error you had mentioned. We could only see server returning 502 statuses which is not Firefox related issue.

Locally I was unable to reproduce the issue using MITM proxy that you had mentioned here.

Could you please provide with more logs from the steps you had mentioned in this comment
We would like logs for both the cases:

  1. When the issue occurs with network.auth.use_redirect_for_retries enabled
  2. Normal working scenario in your with setup network.auth.use_redirect_for_retries disabled
Flags: needinfo?(me+bugzilla)

(In reply to Sunil Mayya from comment #11)

Could you please provide with more logs from the steps you had mentioned in this comment
We would like logs for both the cases:

  1. When the issue occurs with network.auth.use_redirect_for_retries enabled
  2. Normal working scenario in your with setup network.auth.use_redirect_for_retries disabled

Unfortunately, I am unable to provide logs from that computer due to company security policy. If I have time, I might try to reproduce the problem on my own hardware. I think the key might be to set up some authenticated proxy, possibly using Kerberos/GSSAPI or similar, and possibly involving some PAC file.

The logs I've provided so far probably do not demonstrate this specific problem - sorry about that. I haven't figured out a way to artificially reproduce the issues I was experiencing on my personal computer in the absence of a corporate proxy.

Flags: needinfo?(me+bugzilla)
See Also: → 1865968, 1871533
Whiteboard: [necko-triaged][necko-priority-queue] → [necko-triaged][necko-priority-next]

Hello Andrew,
We have fixed a similar issue in https://bugzilla.mozilla.org/show_bug.cgi?id=1871533.
Could you please confirm if the issue is fixed in your company's network/computer in latest nightly build -> https://nightly.mozilla.org ?

Flags: needinfo?(me+bugzilla)

(In reply to Sunil Mayya from comment #13)

Could you please confirm if the issue is fixed in your company's network/computer in latest nightly build -> https://nightly.mozilla.org ?

I just tested nightly build 2024-02-11-21-36-57 on the company laptop. The issue appears to be fixed.

Flags: needinfo?(me+bugzilla)

Great News! Thank you for confirming and your support for all our queries.
I will be resolving this bug as fixed.

Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Duplicate of bug: 1871533
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.