Closed Bug 2092532 Opened 5 days ago Closed 5 days ago

WebTransport datagram write() promises never settle once neqo queues a datagram above its high water mark, stalling datagrams.writable

Categories

(Core :: Networking: HTTP, defect)

Firefox 160
defect

Tracking

()

RESOLVED FIXED
162 Branch
Tracking Status
firefox-esr115 --- unaffected
firefox-esr153 --- unaffected
firefox160 + fixed
firefox161 + fixed
firefox162 + fixed

People

(Reporter: jesup, Assigned: jesup, NeedInfo)

References

(Regression)

Details

(Keywords: regression, Whiteboard: [necko-triaged], [wptsync upstream error])

Attachments

(6 external files)

Reported in https://github.com/mozilla/neqo/issues/4306 against Firefox 160.0 (neqo 0.33.0).

Http3Session::SendDatagram settles a datagram write immediately only when neqo's enqueue returns DatagramQueueOutcome::Ok. For any other outcome (MaxBufferedReached, Overflowed, Rejected) it waits for a per-datagram OutgoingDatagramOutcome, which never arrives:

  • Http3Client drops ConnectionEvent::OutgoingDatagramOutcome, so no outcome reaches the glue.
  • neqo has no outcome at all for a queued datagram that later expires or is evicted, and a Rejected datagram is never queued.

Once outgoingMaxBufferedDatagrams (default 5) such writes are outstanding, content withholds every further write, so datagrams.writable accepts nothing for the rest of the session. The page's writer.ready stays pending; incoming datagrams keep arriving.

Steps to reproduce (from the GitHub report):

  1. Run the aioquic sink (server.py in the issue) and serve the index.html page from the issue over a local HTTP server.
  2. Open the printed URL; the page writes maxDatagramSize-4 byte datagrams as fast as writer.ready allows.

Actual: within about 0.1s a write never settles (5 of 5 unpaced runs over loopback, around write #250, where neqo's 256 KiB queue budget fills). Paced at 140-300 Mbit/s against a remote server it also stalls.
Expected: every write settles, as https://w3c.github.io/webtransport/#webtransportdatagramswritable requires; dropped datagrams free their write.

The real fix is the credit-grant series in bug 1818883, which sends datagrams untracked and paces write() against neqo's queue capacity, but it depends on APIs new in neqo 0.34. For 160 a Gecko-only stopgap settles every write once neqo accepts the datagram, and reports a Rejected datagram as dropped (DROPPED_QUEUE_FULL), with a WPT that every write() of a burst overrunning the outgoing queue settles.

Related: bug 2084924 (same symptom when the session closes concurrently), bug 2083166 (datagram backpressure).

neqo reports no outcome for a queued datagram that later expires or is
evicted, and Http3Client drops the outcomes it does report, so a write()
waiting for one never settled and the datagram writable stalled for the
rest of the session. Settle every write once neqo has accepted the
datagram, and report a datagram neqo refuses as dropped.

Pushed by rjesup@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/50de4adde523 https://hg.mozilla.org/integration/autoland/rev/e04bcc8b9ab2 settle WebTransport datagram writes when neqo queues them above its high water mark r=valentin,necko-reviewers https://github.com/mozilla-firefox/firefox/commit/8341cfa1f58d https://hg.mozilla.org/integration/autoland/rev/71e3c94e7d2c test that every write() of a WebTransport datagram burst that overruns the outgoing queue settles r=valentin

@jesup, is this low risk to include in the dot release that builds on Monday?
If so, given it's an S2, could you add beta and release uplift requests now?

Flags: needinfo?(rjesup)

Very low risk, I'll add them

Flags: needinfo?(rjesup)

neqo reports no outcome for a queued datagram that later expires or is
evicted, and Http3Client drops the outcomes it does report, so a write()
waiting for one never settled and the datagram writable stalled for the
rest of the session. Settle every write once neqo has accepted the
datagram, and report a datagram neqo refuses as dropped.

Original Revision: https://phabricator.services.mozilla.com/D342570

Attachment #9682623 - Flags: approval-mozilla-beta?

firefox-beta Uplift Approval Request

  • User impact if declined/Reason for urgency: Regression that breaks datagrams in WebTransport
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: Exceedingly simple patch; low/no usage of datagrams in production websites
  • String changes made/needed?: none
  • Is Android affected?: yes
Attachment #9682624 - Flags: approval-mozilla-beta?

neqo reports no outcome for a queued datagram that later expires or is
evicted, and Http3Client drops the outcomes it does report, so a write()
waiting for one never settled and the datagram writable stalled for the
rest of the session. Settle every write once neqo has accepted the
datagram, and report a datagram neqo refuses as dropped.

Original Revision: https://phabricator.services.mozilla.com/D342570

Attachment #9682625 - Flags: approval-mozilla-release?

firefox-release Uplift Approval Request

  • User impact if declined/Reason for urgency: Regression that breaks datagrams in WebTransport
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: Exceedingly simple patch; low/no usage of datagrams in production websites
  • String changes made/needed?: none
  • Is Android affected?: yes
Attachment #9682626 - Flags: approval-mozilla-release?
Failed to create upstream wpt PR due to merge conflicts. This requires fixup from a wpt sync admin.
Flags: needinfo?(james)
Flags: needinfo?(aborovova)
Whiteboard: [necko-triaged] → [necko-triaged], [wptsync upstream error]
Status: NEW → RESOLVED
Closed: 5 days ago
Resolution: --- → FIXED
Target Milestone: --- → 162 Branch
Attachment #9682624 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9682623 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #9682626 - Flags: approval-mozilla-release? → approval-mozilla-release+
Attachment #9682625 - Flags: approval-mozilla-release? → approval-mozilla-release+

(In reply to Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org) from comment #12)

Failed to create upstream wpt PR due to merge conflicts. This requires fixup
from a wpt sync admin.

Creating the upstream PR requires https://github.com/web-platform-tests/wpt/pull/64272 to be merged first.

QA Whiteboard: [qa-triage-done-c162/b161]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: