1997372 - Potential regression: http/3 requests failing sporadically on some google backends

Status: UNCONFIRMED

(Core :: Networking: HTTP, defect, P3)

Description

[Olaf Otto]

Attachment: QUIC debug log of an affected session, Single page load + Adjacent resource + XHR loads

Since about a week, our users on both Windows and Linux platform reported that one of our GCP hosted Grafana Platforms became extremely slow and failed to load some graph data on almost every page load.

Our investigation revealed that this only happens in Firefox. Here, Firefox connects to the GCP backend using QUIC (HTTP/3), and performs a number of XHR POST requests to a HTTPS backend. These requests cause queries on the DB that will be answered with some delay (0.5-3 seconds) at most. However, in Firefox, some of these POST requests block, remain without progress for 5-10 seconds, then show up as with NS_ERROR_NET_RESET. The respective requests do not show up in the load balancer logs.

When deactivating HTTP/3 (about:config -> network.http.http3.enable = false), the problem can no longer be reproduced. I suspect this is a regression of https://bugzilla.mozilla.org/show_bug.cgi?id=1734198.

I have enabled QUIC logging, the affected conversation is attached.

Comment 1

[Olaf Otto]

Attachment: dev tool network view of the issue.png

DEV tools network analysis view of the effect of the issue

Comment 2

[Sunil Mayya]

Thanks Olaf for filing the bug and qlogs.
Could you kindly capture HTTP3 networking logs please for this issue as well as this will speed up the investigation.
Go to about:logging and select HTTP/3 preset and logging to a file. If you are concerned about privacy kindly share the logs to [e-mail]
Thanks

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:jesup, since the bug has recent activity, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[[:Propheticus]]

To add a data point. From Zen users we've also received several reports of very slow page loads since the release based on FF 145. Especially on services like Youtube and Google search. The issues don't exist on releases built on FF 144.0.2

In all of the reported cases setting network.http.http3.enable false resolved the issue.
From interacting more in depth with 1 user it turns out they have AdGuard for Windows (desktop app) installed. Disabling/removing it also resolved the issue for them. (but they prefer the disabling of http/3 to be able to continue using AdGuard).

Comment 5

[[:Propheticus]]

p.s. the issue on our Github contains a link to a profiler run in a new profile without extensions: https://share.firefox.dev/4463d80

It shows a lot of NS_ERROR_ABORT and NS_BINDING_ABORTED connections taking 12s up to 30s

• They tested and confirmed the same issue in Firefox
• Downgrading FF to 144.0.2 also solves it

Comment 6

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

The profile does show requests timing out, but because it wasn't captured from about:logging it doesn't show us all the relevant data about the internal state of the networking stack. Could you ask the user to capture a profile from about:logging and then upload it?
Thanks!

Comment 7

[Valentin Gosu [:valentin] {{ PTO 3 July - 3 August }}]

It's possible bug 2001618 mitigates this issue. It would be great if someone could check and confirm.