2063452 - Firefox fails to load some sites with network.http.happy_eyeballs_enabled=true on Zscaler proxy

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P1)

Description

[Sam Johnson]

My company laptop proxies internet traffic using Zscaler. On recent Nightlies (~past month or so), I have been having issues loading sites on Google domains, such as Google Drive and YouTube. The network requests seem to hang (i.e. the browser status bar says "Looked up <domain>" or "Read <domain>") and I need to reload the page for it to fully load. When disabling network.http.happy_eyeballs_enabled, everything starts working as expected. Re-enabling the pref causes these sites to not always load properly again.

I believe this is specific to the Zscaler proxy because my personal machines on the same network have zero issues with network.http.happy_eyeballs_enabled.

What logging can I provide to help troubleshoot this? I will also try to get a mozregression if possible.

Comment 1

[Kershaw Chang [:kershaw]]

Hi,

Could you try to capture a http log?
You could send the log to [e-mail], since it may contain some personal information.

Thanks.

Comment 2

[Sam Johnson]

(In reply to Kershaw Chang [:kershaw] from comment #1)

Hi,

Could you try to capture a http log?
You could send the log to [e-mail], since it may contain some personal information.

Thanks.

Apologies for the delay! I have sent the logs via email. The logs include a gemini.google.com tab which never fully loaded. Please let me know if there are additional captures which would be helpful.

Comment 3

[Kershaw Chang [:kershaw]]

Thanks for the log. I think I've figured out what's happening.

1. AltSvcCache::UpdateAltServiceMapping starts eager validation of an h3 alternative and marks the connection info as Http3Only. This prevents Happy Eyeballs from falling back to H1/H2 and incorrectly validating the mapping — AlternateServices.cpp#937.

2. CreateHappyEyeballs respects that flag and initializes the engine with h1 = false, h2 = false — HappyEyeballsConnectionAttempt.cpp#197.

3. This is correct as long as the attempt remains speculative. The problem is that the speculative connection can be also used by normal transaction and inherit a race that can never fall back to TCP. As a result, users with UDP blocked can never reach sites that support HTTP/3.

Since we've already let Happy Eyeballs ride the trains in bug 2062892, I think the most important thing to do right now is to revert that change to avoid affecting release users.

Comment 4

[Valentin Gosu [:valentin]]

Attachment: Bug 2063452 - test r=#necko

Comment 5

[Donal Meehan [:dmeehan]]

Bug 2067488 landed in main and beta

Comment 6

[Kershaw Chang [:kershaw]]

Attachment: Bug 2063452 - Give h3-only connection infos their own connection entry, r=mxinden,#necko

Eager Alt-Svc H3 validation marks its connection info as Http3Only, so Happy Eyeballs races H3 without a TCP fallback. Previously, this attempt shared the origin's ConnectionEntry, allowing a normal transaction to claim a race that could never fall back to TCP. At the same time, the unconnected UDP attempt kept HasActiveH3Connection() true, causing AtActiveConnectionLimit() to reject any new connections to the origin. When UDP was blocked, the origin was effectively pinned to this single connection attempt until it was reaped.

This patch combines Http3Disabled and Http3Only into a single Http3Policy field and gives H3-only connection info a separate hash key ([h3only], alongside the existing [!h3]). This isolates H3-only attempts in their own connection entry. Once an H3 connection is successfully established, it is handed over to the origin's regular connection entry.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e72064d9c929
https://hg.mozilla.org/integration/autoland/rev/45e24e6abbfc
Give h3-only connection infos their own connection entry, r=mxinden,necko-reviewers,valentin

Comment 8

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/45e24e6abbfc