2044951 - With DoH, HEv3 races unreachable IPv6 first on cold connections, adding up to 40ms depending on the platform

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect, P2)

Description

[Andrew Creskey [:acreskey]]

This is what I'm observing on IPV4 only networks with DoH enabled, MacOS, and HeV3:

1. We attempt to connect by QUIC over IPV6 first (even though Firefox's connectivity service has failed to detect an ipv6 path nsHttp — [Verbose] nsHttp: 134826b60 prefer ipv4=1, ipv6=0)
2. Black hole -- we wait for 250ms, happy_eyeballs_connection_attempt_delay
3. We then try QUIC over IPV4, which succeeds
4. Once the IPV4 connection succeeds, we abort the IPv6 connection attempt
5. Example: cloudflare: https://share.firefox.dev/4ocLHYQ
6. And so it looks like every cold HTTP3 connection ends up being delayed by 250ms as we try IPV6 first
7. Example, from the same session: www.google.com : https://share.firefox.dev/4g0unUx

Comment 1

[Andrew Creskey [:acreskey]]

Attachment: time_to_request_start_doh.png

Comment 2

[Andrew Creskey [:acreskey]]

There is possible evidence from pageload event that may reflect a delay in time_to_request_start over the desktop platforms since HEV3 was introduced.

https://sql.telemetry.mozilla.org/queries/120953/source?p_days=180&p_metric=time_to_request_start&p_percentile=75&p_product=firefox_desktop.pageload#295067

But this is observational and I'm also seeing some upticks in some cases with hev3 and the os resolver.

Comment 3

[Andrew Creskey [:acreskey]]

On Fenix Nightly, on a network with no iPV6
nsHttp — [Verbose] nsHttp: 718083fa30 prefer ipv4=1, ipv6=0

We try quic IPV6 first but fail back relatively quickly, about 36ms after the attempt for the cloudflare connect:

http://www.cloudflare.com

5.481s nsHttp — [Info] nsHttp: Http3Session::Init origin=www.cloudflare.com, alpn=h3, selfAddr=::, peerAddr=2606:4700::6810:7c60, qpack table size=65536, max blocked streams=20 webtransport=0 [this=70c27afa00]
5.517s nsHttp — [Info] nsHttp: Http3Session::Init origin=www.cloudflare.com, alpn=h3, selfAddr=0.0.0.0, peerAddr=104.16.124.96, qpack table size=65536, max blocked streams=20 webtransport=0 [this=7032bdf600]

https://share.firefox.dev/49Hjvaj

---

On www.google.com, we wait 39ms for the IPV6 connect to fail:

11.232s nsHttp — [Info] nsHttp: Http3Session::Init origin=www.google.com, alpn=h3, selfAddr=::, peerAddr=2001:4860:482a:7700::, qpack table ,,,
11.271s nsHttp — [Info] nsHttp: Http3Session::Init origin=www.google.com, alpn=h3, selfAddr=0.0.0.0, peerAddr=142.251.155.119, qpack table size=65536, max blocked streams=20 webtransport=0 [this=6fe43d4700]

https://share.firefox.dev/4fr37P4

Comment 4

[Andrew Creskey [:acreskey]]

I ran a live site Linux navigation in CI
(IPV4 only)

There are moz_logs stored with the artifacts

We try www.cloudflare.com via IPV6 first.

2026-06-04 19:16:25.062043 UTC - [Parent 6034: Socket Thread]: I/nsHttp Http3Session::Init origin=www.cloudflare.com, alpn=h3, selfAddr=::, peerAddr=2606:4700::6810:7c60, qpack table size=65536, max blocked streams=20 webtransport=0 [this=723454988600]

and 2ms later we fallback to IPV4

2026-06-04 19:16:25.064377 UTC - [Parent 6034: Socket Thread]: I/nsHttp Http3Session::Init origin=www.cloudflare.com, alpn=h3, selfAddr=0.0.0.0, peerAddr=104.16.124.96, qpack table size=65536, max blocked streams=20 webtransport=0 [this=723454988600]

However on the www.google.com connection, I'm seeing a delay of 32ms between the initial IPV6 attempt and the IPV4 attempt (which ultimately succeed)

2026-06-04 19:16:33.057473 UTC - [Parent 6034: Socket Thread]: I/nsHttp Http3Session::Init origin=www.google.com, alpn=h3, selfAddr=::, peerAddr=2001:4860:4828:7700::, qpack table size=65536, max blocked streams=20 webtransport=0 [this=723431430400]
...
2026-06-04 19:16:33.058404 UTC - [Parent 6034: Socket Thread]: I/nsHttp Http3Session::Init origin=www.google.com, alpn=h3, selfAddr=::, peerAddr=2001:4860:4828:7700::, qpack table size=65536, max blocked streams=20 webtransport=0 [this=723430e2ba00]
2026-06-04 19:16:33.090484 UTC - [Parent 6034: Socket Thread]: V/nsHttp HappyEyeballsConnectionAttempt::ProcessConnectionResult 7234405fc4e0 addr=[2001:4860:4828:7700::] id=3 aStatus=804b0014

this fails 32ms later 804b0014, NS_ERROR_NET_RESET, and we try again on IPV4 and succeed:

2026-06-04 19:16:33.090685 UTC - [Parent 6034: Socket Thread]: I/nsHttp Http3Session::Init origin=www.google.com, alpn=h3, selfAddr=0.0.0.0, peerAddr=142.251.150.119, qpack table size=65536, max blocked streams=20 webtransport=0 [this=723431430700]

2026-06-04 19:16:33.091171 UTC - [Parent 6034: Socket Thread]: V/nsHttp ConnectionEstablisher::ActivateConnectionWithTransaction 7234371033a0 conn=72343031e000 trans=72343aa5abd0
2026-06-04 19:16:33.091177 UTC - [Parent 6034: Socket Thread]: E/nsHttp HttpConnectionUDP::Activate [this=72343031e000 trans=72343aa5abd0 caps=401]
2026-06-04 19:16:33.091207 UTC - [Parent 6034: Socket Thread]: V/nsHttp HttpConnectionUDP::ResumeSend [this=72343031e000]
...
2026-06-04 19:16:33.154459 UTC - [Parent 6034: Socket Thread]: V/nsHttp HappyEyeballsConnectionAttempt::ProcessConnectionResult 7234405fc4e0 addr=[142.251.150.119] id=4 aStatus=0

Comment 5

[Andrew Creskey [:acreskey]]

Looking at MacOs in CI
(https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DeF-Z5aXRkG5HTv0iXflzQ/runs/0/artifacts/public/test_info/browsertime-results-extra.tgz)

Most all of the IPV6 connects fail within a millisecond or so.

However I can see some of the connections are delayed a bit, e.g. google.com here with 14ms waiting for the doomed IPV6 connection to fail before we start the IPV4 one:

2026-06-04 19:30:44.526649 UTC - [Parent 1769: Socket Thread]: I/nsHttp Http3Session::Init origin=www.google.com, alpn=h3, selfAddr=::, peerAddr=2001:4860:4827:7700::, qpack table size=65536, max blocked streams=20 webtransport=0 [this=14e70a100]

2026-06-04 19:30:44.540681 UTC - [Parent 1769: Socket Thread]: I/nsHttp Http3Session::Init origin=www.google.com, alpn=h3, selfAddr=0.0.0.0, peerAddr=142.251.152.119, qpack table size=65536, max blocked streams=20 webtransport=0 [this=14e737f00]

The 250ms delay that I'm seeing on my own Mac is almost certainly a biproduct of the network interface filters that the Crowdstrike monitoring software installs, as Kershaw pointed out.
(So renaming the bug)

Comment 6

[Andrew Creskey [:acreskey]]

On the Windows try push:

Artifacts

In general these look like the fastest IPV6 fails:

e.g., 0ms fast fail connecting to www.cloudflare.com,

2026-06-04 19:44:49.135000 UTC - [Parent 9056: Socket Thread]: I/nsHttp Http3Session::Init origin=www.cloudflare.com, alpn=h3, selfAddr=::, peerAddr=2606:4700::6810:7c60, qpack table size=65536, max blocked streams=20 webtransport=0 [this=165ed9dfa00]
...
2026-06-04 19:44:49.135000 UTC - [Parent 9056: Socket Thread]: V/nsHttp HttpConnectionUDP::CloseTransaction[this=165e6bbc400 trans=165ed9dfa08 reason=804b0014]
...
2026-06-04 19:44:49.135000 UTC - [Parent 9056: Socket Thread]: I/nsHttp Http3Session::Init origin=www.cloudflare.com, alpn=h3, selfAddr=0.0.0.0, peerAddr=104.16.124.96, qpack table size=65536, max blocked streams=20 webtransport=0 [this=165ed9e2d00]

Although there are occasional connections that are delayed.
e.g. 21ms delay on the connection to fonts.gstatic.com made for discord.com

2026-06-04 19:45:26.460000 UTC - [Parent 9056: Socket Thread]: I/nsHttp Http3Session::Init origin=fonts.gstatic.com, alpn=h3, selfAddr=::, peerAddr=2607:f8b0:4005:809::2003, qpack table size=65536, max blocked streams=20 webtransport=0 [this=165eabf3d00]
...
2026-06-04 19:45:26.481000 UTC - [Parent 9056: Socket Thread]: I/nsHttp Http3Session::Init origin=fonts.gstatic.com, alpn=h3, selfAddr=0.0.0.0, peerAddr=142.251.218.163, qpack table size=65536, max blocked streams=20 webtransport=0 [this=165f0d7e100]

Comment 7

[Max Inden]

Thank you Andrew. Indication that the high delays only occur with Crowdstrike (on MacOS) is helpful.

On www.google.com, we wait 39ms for the IPV6 connect to fail:

However on the www.google.com connection, I'm seeing a delay of 32ms between the initial IPV6 attempt and the IPV4 attempt (which ultimately succeed)

However I can see some of the connections are delayed a bit, e.g. google.com here with 14ms waiting for the doomed IPV6 connection to fail before we start the IPV4 one:

Although there are occasional connections that are delayed. e.g. 21ms delay on the connection to fonts.gstatic.com made for discord.com

Judging by the values themselves (i.e. being below 250ms), I assume each of these are due to the in process overhead of the connection attempt itself, i.e. the time Firefox needs to start the attempt, not directly related to Happy Eyeballs v3. That said, we might want to re-introduce a heuristic, to cut down on that extra delay, e.g. if the last 4 connections have been IPv4, start with IPv4 first, i.e. set DualStackPreferIpv4.

Comment 8

[Valentin Gosu [:valentin]]

if the last 4 connections have been IPv4, start with IPv4 first, i.e. set DualStackPreferIpv4.

We do have a ConnectivityService which can tell if you have DNSv6/IPv6 conectivity to our captive portal endpoint. We could use that as a signal.

Comment 9

[Max Inden]

Attachment: Bug 2044951 - Prefer IPv4 when connectivity service reports no IPv6

On a cold HTTP connection there is no per-origin address family preference
yet, so Happy Eyeballs v3 races IPv6 first by default. On IPv4-only networks
with DoH enabled, AAAA records still resolve, so we race a doomed IPv6 attempt
and only fall back to IPv4 after it fails, adding latency to every cold HTTP/3
connection.

Reuse the existing NetworkConnectivityService signal: when it has determined
that IPv6 is unreachable on the current network while IPv4 works, start with
IPv4 (DualStackPreferV4) instead of IPv6. IPv6 remains a delayed fallback, so
a stale or wrong probe only costs a delayed attempt rather than a hard failure.

Comment 10

[Max Inden]

I drafted Valentin's suggestion in https://phabricator.services.mozilla.com/D305586. Thoughts?

Comment 11

[Andrew Creskey [:acreskey]]

(In reply to Max Inden from comment #10)

I drafted Valentin's suggestion in https://phabricator.services.mozilla.com/D305586. Thoughts?

This patch is similar in action to what Claude wrote up last week - and that one resolved the exaggerated 250ms delay I was seeing due to Crowdstrike monitoring software.

So I'm +1 on this general direction and if this can be landed I'm happy to monitor telemetry and re-run the above try pushes to give us a glimpse of performance over our four platforms.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/4922337e38a3
https://hg.mozilla.org/integration/autoland/rev/2f9db9ca5f65
Prefer IPv4 when connectivity service reports no IPv6 r=necko-reviewers,kershaw

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/e7e1e5027eb7
https://hg.mozilla.org/integration/autoland/rev/ce4436149b16
Revert "Bug 2044951 - Prefer IPv4 when connectivity service reports no IPv6 r=necko-reviewers,kershaw" for causing mass failures @nsISupportsImpl.cpp.

Comment 14

[agoloman]

Backed out for causing mass failures @nsISupportsImpl.cpp.

• Backout link
• Push with failures
• Failure Log

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/c5ea1c914628
https://hg.mozilla.org/integration/autoland/rev/c1bca6affdee
Prefer IPv4 when connectivity service reports no IPv6 r=necko-reviewers,kershaw

Comment 16

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/c1bca6affdee

Comment 17

[Andrew Creskey [:acreskey]]

I'll have more to share, but I'm seeing this change work very nicely.

Can see the connections trying IPV4 first
https://share.firefox.dev/4xDrEGY

Fun fact -- our automation stack disables the connectivity service, network.connectivity-service.enabled in no fewer than three places, so currently this change is not apparent in CI.

- browsertime (bundled prefs): tools/browsertime/node_modules/browsertime/lib/firefox/settings/firefoxPreferences.js:184 → false
- geckodriver (embedded remote agent): remote/shared/RecommendedPreferences.sys.mjs:356 → false ("Disable connectivity service pings")
- marionette: testing/marionette/client/marionette_driver/geckoinstance.py:162 → False

(will be logging this automation change shortly)