1979279 - High packet loss on QUIC on Windows with GSO

Status: ASSIGNED

(Core :: Networking, defect, P2)

Description

[Max Inden]

Seeing high packet loss on QUIC connections on Windows since Neqo v0.14.0 landed.

See following graph. Note sharp uptick in 95th percentile on 2025-07-14 from ~0.5% packet loss to ~6.5% packet loss, same day as Bug 1975873 landed.

https://glam.telemetry.mozilla.org/firefox/probe/http3_loss_ratio/explore?os=Windows&process=parent&visiblePercentiles=%5B95%2C75%2C50%5D

Comment 1

[Max Inden]

Attachment: image.png

Comment 2

[Max Inden]

Hypothesis thus far:

• Bug 1975873 enables USO on Windows
• on the 95th percentile, Firefox on Windows sends 2 or more datagrams per WSA_SENDMSG sys call
• since Firefox now uses hybrid post quantum TLS handshake, the first client flight has two UDP datagrams
• thus Firefox UDP QUIC connections do at least one USO WSA_SENDMSG call with 2 or more datagrams per connection
• it is unlikely that both get lost, as otherwise the loss ratio would not be at ~6.5% but 100%
• instead, the assumption is that the first is sent, but the second is lost

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1975873

Comment 4

[Max Inden]

Attachment: Bug 1979279 - disable QUIC GSO on Windows r=kershaw

Disable HTTP3 QUIC UDP GSO IO, i.e. sending multiple UDP datagrams in
one sys call, on Windows. See Bug for details.

Comment 5

[Max Inden]

I suggest the following:

1. disable GSO (aka. USO) on Windows on Firefox Nightly (see patch above)
2. if it fixes the issue:
   1. backport the fix to Beta
   2. investigate why GSO is failing on (some) Windows devices
3. if it doesn't fix the issue:
   1. find a new hypothesis

Comment 6

[Kershaw Chang [:kershaw]]

FYI: GSO might also be the root cause of bug 1978821.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/f0c091327c40
https://hg.mozilla.org/integration/autoland/rev/5bd98c9ca0fd
disable QUIC GSO on Windows r=kershaw

Comment 9

[Max Inden]

Potential fix landed. Once metrics on Nightly are back to normal, I will propose a back-port for Beta.

Comment 10

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/5bd98c9ca0fd

Comment 11

[Max Inden]

Latest data point on Glam is 2025-07-24. Thus, there is no confirmation (or lack of confirmation) whether the above patch fixed the issue. Will check back in tomorrow.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The bug is marked as tracked for firefox142 (beta). However, the bug still has low severity.

:ghess, could you please increase the severity for this tracked bug? If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Comment 13

[Max Inden]

Attachment: Bug 1979279 - disable QUIC GSO on Windows r=kershaw

Disable HTTP3 QUIC UDP GSO IO, i.e. sending multiple UDP datagrams in
one sys call, on Windows. See Bug for details.

Original Revision: https://phabricator.services.mozilla.com/D258686

Comment 14

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined: Degraded network performance
• Code covered by automated testing: yes
• Fix verified in Nightly: yes
• Needs manual QE test: no
• Steps to reproduce for manual QE testing: no manual QE testing
• Risk associated with taking this patch: Not aware of additional risks. Disables a new feature (i.e. GSO)
• Explanation of risk level: Not sure how to clasify.
• String changes made/needed: no
• Is Android affected?: no

Comment 15

[Max Inden]

Attachment: image.png

Status update:

• since the patch to Nightly, the loss_ratio metric on Nightly has recovered
• thus I requested a Beta uplift https://phabricator.services.mozilla.com/D258916
• I tried to reproduce the bug on 6 different Windows machines (x86-64 and ARM) without success

Comment 16

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/19f243d2de03
https://hg.mozilla.org/releases/mozilla-beta/rev/3414a7c5a3d1