Closed Bug 1734198 Opened 4 months ago Closed 3 months ago

http/3 requests failing sporadically on some google backends

Categories

(Core :: Networking: HTTP, defect, P2)

Firefox 92
defect

Tracking

()

RESOLVED FIXED
95 Branch
Tracking Status
firefox95 --- fixed

People

(Reporter: gaetanhervouet, Assigned: dragana)

References

(Blocks 1 open bug)

Details

(Whiteboard: [necko-triaged])

Attachments

(2 files)

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.61 Safari/537.36

Steps to reproduce:

Actual results:

We are serving dash video stream MPDs from dai.google.com and the dash segments requests first go to dai.google.com before being redirected to *.googlevideo.com.
Most of the time, the segment requests like 0.mp4 and its redirect to googlevideo works properly (requests and responses in the paste link):

https://storage.googleapis.com/gaetan/firefox-bug/0.mp4.png

GET /linear/pods/v1/p/PSzZMzAkSXCmlJOWDmRj8Q/7df71a9d-dd2c-4937-97b6-94afda289f9f:MRN2/320530/0/1/eccbc87e4b5ce2fe28308fd9f2a7baf3/0.mp4 HTTP/3
Host: dai.google.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0
Accept: /
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Origin: https://developers-dot-devsite-v2-prod.appspot.com
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
Referer: https://developers-dot-devsite-v2-prod.appspot.com/
Connection: keep-alive
TE: trailers

HTTP/3 302 Found
access-control-allow-credentials: true
access-control-allow-headers: Authorization
access-control-allow-origin: https://developers-dot-devsite-v2-prod.appspot.com
access-control-expose-headers: Location
cache-control: no-cache, no-store, max-age=0, must-revalidate
content-type:
content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8
date: Tue, 05 Oct 2021 14:24:16 GMT
expires: Mon, 01 Jan 1990 00:00:00 GMT
location: https://r3---sn-gvbxgn-tt1e7.googlevideo.com/videoplayback?expire=1633530253&ei=DWBcYc6tHtbGhwbTpqCoCg&ip=0.0.0.0&goc=25&faskm=469&source=dclk_video_ads&acao=yes&ctier=L&id=4f45885ae05aaf91&itag=340&nc=21775744923&mime=audio/mp4&sgoap=gir%3Dyes%3Bitag%3D340%3Bxtags%3Dac%3Daac%3Aaccf%3Dlc%3Aach%3D2%3Aasr%3D48000%3Aatbr%3D128%3Bclen%3D160210%3Bdur%3D10047%3Blmt%3D1629911834958682&requiressl=yes&susc=dvc&sparams=expire,ei,ip,goc,faskm,source,acao,ctier,id,itag,mime,sgoap,requiressl,susc&sig=AOq0QJ8wRQIhALlcPUTOFGuueDkhLECLQEzTqwvNxSDYwOYxs4HtqpBmAiAgtpe0HZmSh8_MXitsSxf_cw5Q0TLx911Kk5HjKSZ84Q==&initcwndbps=8980&mh=kE&mip=99.229.72.144&mm=31&mn=sn-gvbxgn-tt1e7&ms=au&mt=1633443688&mv=m&mvi=3&pcm2cms=yes&pl=17&lsparams=initcwndbps,mh,mip,mm,mn,ms,mv,mvi,pcm2cms,pl&lsig=AG3C_xAwRQIhAJonml-F9GqZMGhpsefTmy0sYBF2YZfxrYgaqoKbOP6YAiB6fRabuYaviLQIpLsxAVprI6YO5eAgTEiw5Cw-J_zVfA%3D%3D&faudshow=94&faudskip=0&goap=slices%3D0-159652&cpn=ffcand0sSTeXtpSv
pragma: no-cache
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 0
content-length: 0
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

Redirection:
https://storage.googleapis.com/gaetan/firefox-bug/0.mp4.redirect.png

GET /videoplayback?expire=1633530253&ei=DWBcYc6tHtbGhwbTpqCoCg&ip=0.0.0.0&goc=25&faskm=469&source=dclk_video_ads&acao=yes&ctier=L&id=4f45885ae05aaf91&itag=340&nc=21775744923&mime=audio/mp4&sgoap=gir%3Dyes%3Bitag%3D340%3Bxtags%3Dac%3Daac%3Aaccf%3Dlc%3Aach%3D2%3Aasr%3D48000%3Aatbr%3D128%3Bclen%3D160210%3Bdur%3D10047%3Blmt%3D1629911834958682&requiressl=yes&susc=dvc&sparams=expire,ei,ip,goc,faskm,source,acao,ctier,id,itag,mime,sgoap,requiressl,susc&sig=AOq0QJ8wRQIhALlcPUTOFGuueDkhLECLQEzTqwvNxSDYwOYxs4HtqpBmAiAgtpe0HZmSh8_MXitsSxf_cw5Q0TLx911Kk5HjKSZ84Q==&initcwndbps=8980&mh=kE&mip=99.229.72.144&mm=31&mn=sn-gvbxgn-tt1e7&ms=au&mt=1633443688&mv=m&mvi=3&pcm2cms=yes&pl=17&lsparams=initcwndbps,mh,mip,mm,mn,ms,mv,mvi,pcm2cms,pl&lsig=AG3C_xAwRQIhAJonml-F9GqZMGhpsefTmy0sYBF2YZfxrYgaqoKbOP6YAiB6fRabuYaviLQIpLsxAVprI6YO5eAgTEiw5Cw-J_zVfA%3D%3D&faudshow=94&faudskip=0&goap=slices%3D0-159652&cpn=ffcand0sSTeXtpSv HTTP/3
Host: r3---sn-gvbxgn-tt1e7.googlevideo.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0
Accept: /
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Origin: null
Referer: https://developers-dot-devsite-v2-prod.appspot.com/
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
TE: trailers

HTTP/3 200 OK
last-modified: Wed, 25 Aug 2021 17:17:14 GMT
content-type: audio/mp4
date: Tue, 05 Oct 2021 14:24:16 GMT
expires: Tue, 05 Oct 2021 14:24:16 GMT
cache-control: private, max-age=86097
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
client-protocol: quic
access-control-allow-origin: null
access-control-allow-credentials: true
timing-allow-origin: null
access-control-expose-headers: Client-Protocol, Content-Length, Content-Type, X-Bandwidth-Est, X-Bandwidth-Est2, X-Bandwidth-Est3, X-Bandwidth-App-Limited, X-Bandwidth-Est-App-Limited, X-Bandwidth-Est-Comp, X-Bandwidth-Avg, X-Head-Time-Millis, X-Head-Time-Sec, X-Head-Seqnum, X-Response-Itag, X-Restrict-Formats-Hint, X-Sequence-Num, X-Segment-Lmt, X-Walltime-Ms
vary: Origin
cross-origin-resource-policy: cross-origin
x-content-type-options: nosniff
server: gvs 1.0

At some point though, few of the same requests start failing:
Get 0.mp4 has the same CORS headers as previously but firefox reports it as an error.
https://storage.googleapis.com/gaetan/firefox-bug/0.mp4.failed.png

GET /linear/pods/v1/p/PSzZMzAkSXCmlJOWDmRj8Q/7df71a9d-dd2c-4937-97b6-94afda289f9f:MRN2/320542/0/0/eccbc87e4b5ce2fe28308fd9f2a7baf3/0.mp4 HTTP/3
Host: dai.google.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0
Accept: /
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Origin: https://developers-dot-devsite-v2-prod.appspot.com
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
Referer: https://developers-dot-devsite-v2-prod.appspot.com/
Connection: keep-alive
TE: trailers

HTTP/3 302 Found
access-control-allow-credentials: true
access-control-allow-headers: Authorization
access-control-allow-origin: https://developers-dot-devsite-v2-prod.appspot.com
access-control-expose-headers: Location
cache-control: no-cache, no-store, max-age=0, must-revalidate
content-type:
content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8
date: Tue, 05 Oct 2021 14:36:00 GMT
expires: Mon, 01 Jan 1990 00:00:00 GMT
location: https://r3---sn-gvbxgn-tt1e7.googlevideo.com/videoplayback?expire=1633530955&ei=y2JcYbrQAoaa8wTNnJXwAQ&ip=0.0.0.0&nc=21775744923&sgoap=gir%3Dyes%3Bitag%3D340%3Bxtags%3Dac%3Daac%3Aaccf%3Dlc%3Aach%3D2%3Aasr%3D48000%3Aatbr%3D128%3Bclen%3D160210%3Bdur%3D10047%3Blmt%3D1629911834958682&requiressl=yes&id=4f45885ae05aaf91&source=dclk_video_ads&itag=340&mime=audio/mp4&goc=25&faskm=469&acao=yes&ctier=L&susc=dvc&sparams=expire,ei,ip,sgoap,requiressl,id,source,itag,mime,goc,faskm,acao,ctier,susc&sig=AOq0QJ8wRQIgVl5nB_EdhTPggEY3ybIpCENNs6mIuSN-pyhncT6Di78CIQCkhKqc4vBDwW-bWihWmniwCwzXGWxm8GAZlnHqins18g==&initcwndbps=8820&mh=kE&mip=99.229.72.144&mm=31&mn=sn-gvbxgn-tt1e7&ms=au&mt=1633444392&mv=m&mvi=3&pcm2cms=yes&pl=17&lsparams=initcwndbps,mh,mip,mm,mn,ms,mv,mvi,pcm2cms,pl&lsig=AG3C_xAwRAIgEU7GeT7vc81OoRgjn4VN0FDIzgFL1QaogEEf2BRidfICIAOOy7HwbtdSHbQmc1Aro4lByGO3YRPSpAQYxWAk1gAd&faudshow=94&faudskip=0&goap=slices%3D0-159652&cpn=ffcand0sSTeXtpSv
pragma: no-cache
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 0
content-length: 0
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

The redirected googlevideo URL does not even get a response:
https://storage.googleapis.com/gaetan/firefox-bug/0.mp4.redirect.failed.png
GET /videoplayback?expire=1633530955&ei=y2JcYbrQAoaa8wTNnJXwAQ&ip=0.0.0.0&nc=21775744923&sgoap=gir%3Dyes%3Bitag%3D340%3Bxtags%3Dac%3Daac%3Aaccf%3Dlc%3Aach%3D2%3Aasr%3D48000%3Aatbr%3D128%3Bclen%3D160210%3Bdur%3D10047%3Blmt%3D1629911834958682&requiressl=yes&id=4f45885ae05aaf91&source=dclk_video_ads&itag=340&mime=audio/mp4&goc=25&faskm=469&acao=yes&ctier=L&susc=dvc&sparams=expire,ei,ip,sgoap,requiressl,id,source,itag,mime,goc,faskm,acao,ctier,susc&sig=AOq0QJ8wRQIgVl5nB_EdhTPggEY3ybIpCENNs6mIuSN-pyhncT6Di78CIQCkhKqc4vBDwW-bWihWmniwCwzXGWxm8GAZlnHqins18g==&initcwndbps=8820&mh=kE&mip=99.229.72.144&mm=31&mn=sn-gvbxgn-tt1e7&ms=au&mt=1633444392&mv=m&mvi=3&pcm2cms=yes&pl=17&lsparams=initcwndbps,mh,mip,mm,mn,ms,mv,mvi,pcm2cms,pl&lsig=AG3C_xAwRAIgEU7GeT7vc81OoRgjn4VN0FDIzgFL1QaogEEf2BRidfICIAOOy7HwbtdSHbQmc1Aro4lByGO3YRPSpAQYxWAk1gAd&faudshow=94&faudskip=0&goap=slices%3D0-159652&cpn=ffcand0sSTeXtpSv undefined
Host: r3---sn-gvbxgn-tt1e7.googlevideo.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0
Accept: /
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Origin: null
Referer: https://developers-dot-devsite-v2-prod.appspot.com/
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
TE: trailers

no response

Expected results:

The request should have succeeded like they did just before.
The server response didn't change compared to the previous successful segment requests yet the requests fail.

I have been able to reproduce this issue on both macOSX and Windows 10 x64 firefox versions.

When I go in about:config and set "network.http.http3.enabled" to false, restart firefox and try that page again, the problem stops happening.

In this log file
https://dai.google.com/linear/pods/v1/p/PSzZMzAkSXCmlJOWDmRj8Q/b39216d1-2b5c-48f0-9dab-a168e9b3d242:MRN2/320746/0/0/eccbc87e4b5ce2fe28308fd9f2a7baf3/1.mp4

AND

https://r3---sn-gvbxgn-tt1e7.googlevideo.com/videoplayback?expire=1633543194&ei=mpJcYZ7FGYXa8wS24Y6YCw&ip=0.0.0.0&ctier=L&source=dclk_video_ads&goc=25&faskm=469&requiressl=yes&acao=yes&id=4f45885ae05aaf91&itag=340&nc=21775744923&mime=audio/mp4&sgoap=gir%3Dyes%3Bitag%3D340%3Bxtags%3Dac%3Daac%3Aaccf%3Dlc%3Aach%3D2%3Aasr%3D48000%3Aatbr%3D128%3Bclen%3D160210%3Bdur%3D10047%3Blmt%3D1629911834958682&susc=dvc&sparams=expire,ei,ip,ctier,source,goc,faskm,requiressl,acao,id,itag,mime,sgoap,susc&sig=AOq0QJ8wRgIhAPdgFhe7U86ECcxfQuf3DPOXMwJ7HvdgM3SxOxy9nYpiAiEA4qRN0I5E4G3FcdfzZ7Tkn1Va7CLifWBv9GJnYM6BgKo=&initcwndbps=12110&mh=kE&mip=99.229.72.144&mm=31&mn=sn-gvbxgn-tt1e7&ms=au&mt=1633456149&mv=m&mvi=3&pcm2cms=yes&pl=17&lsparams=initcwndbps,mh,mip,mm,mn,ms,mv,mvi,pcm2cms,pl&lsig=AG3C_xAwRAIgSa9pLa8dX0SlvXOwEaux1Ia6q34fFH7EeEgxxt9J3e4CIBhVAYzJl__231Pp_F7NVrCi8UCfrMwHwiq_T2OiwGrf&faudshow=94&faudskip=94&goap=slices%3D0-159652&cpn=s5IW0StcSPCdq6Fo

failed with the same CORS error from that bug.

https://dai.google.com/linear/pods/v1/p/PSzZMzAkSXCmlJOWDmRj8Q/b39216d1-2b5c-48f0-9dab-a168e9b3d242:MRN2/320746/0/0/c4ca4238a0b923820dcc509a6f75849b/1.mp4

AND

https://r3---sn-gvbxgn-tt1e7.googlevideo.com/videoplayback?expire=1633543194&ei=mpJcYZ7FGYXa8wS24Y6YCw&ip=0.0.0.0&requiressl=yes&source=dclk_video_ads&nc=21775744923&fvskm=300&acao=yes&ctier=L&id=4f45885ae05aaf91&itag=340&sgovp=gir%3Dyes%3Bitag%3D340%3Bxtags%3Dvc%3Davc%3Avccf%3Dhigh%3Avfr%3D59.94%3Avsz%3D1280x720%3Avtbr%3D5400%3Bclen%3D385943%3Bdur%3D9993%3Blmt%3D1629911842971145&mime=video/mp4&goc=25&susc=dvc&sparams=expire,ei,ip,requiressl,source,fvskm,acao,ctier,id,itag,sgovp,mime,goc,susc&sig=AOq0QJ8wRgIhAI5o3xJzaRMeRGbc8SuxbDUYlyI1jzIpNmnvXTxvh7ljAiEAyBDvQ0mSc7lQXaQPMq7yNNFfb8siwhWryCC9NbdWlPs=&initcwndbps=12110&mh=kE&mip=99.229.72.144&mm=31&mn=sn-gvbxgn-tt1e7&ms=au&mt=1633456149&mv=m&mvi=3&pcm2cms=yes&pl=17&lsparams=initcwndbps,mh,mip,mm,mn,ms,mv,mvi,pcm2cms,pl&lsig=AG3C_xAwRQIhAN_MjAgr0uLV18S9t7w6q6103FhKEzIF6AjPk_D8igcKAiB6h5OgI0BHVBOLT_TlPuJMaomLOMBs9Xr6x4lGnhmuSw%3D%3D&fvidshow=120&fvidskip=120&govp=slices%3D0-208615&cpn=s5IW0StcSPCdq6Fo

succeeded

The Bugbug bot thinks this bug should belong to the 'Core::DOM: Security' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Component: Untriaged → DOM: Security
Product: Firefox → Core
Blocks: QUIC
Severity: -- → S3
Component: DOM: Security → Networking: HTTP
Priority: -- → P2

Thank you for the log.
The CORS fails because requests get cancelled after we receive a stateless reset. We should restart such requests if possible (e.g. if a request has already received data we cannot restart it). I will make a fix for this.

I am not sure why we are getting a stateless request. From log it looks like the server is sending after receiving the first 1RTT packet from the client.

  • The handshake is just finished.
  • Firefox send first requests.
  • Firefox get a 1rtt packet from the server, that is decrypted fine (it is data on one of the control or qpack streams).
  • ~15ms later Firefox receives a stateless reset. (probably reaction to the Firefox first 1RTT packet)

The connection uses 0RTT and 0RTT data are rejected by the server.

I will have another look to see if I can find there is something wrong with the firefox packet that is triggering stateless reset.

Thanks Dragana, are you noticing this behavior on the dai.google.com request or on the googlevideo.com request?

It is on googlevideo.com

This is an appropriate error to set and it may cause transactions to be restarted if possible (e.g. no data has been received already and the method is safe).

Assignee: nobody → dd.mozilla

Just as a note here, though it doesn't read on the specific problem... When I tried this on my browser, it failed for other reasons. I have strict tracking protection enabled, so I got this message: "Failed to get service worker registration(s): Storage access is restricted in this context due to user settings or private browsing mode." I've flagged this with the team that works on these features so they are aware.

Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Whiteboard: [necko-triaged]
Pushed by ddamjanovic@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a8c7644c02d0
Close transactions with NS_ERROR_NET_RESET if a stateless reset is received on a HTTP/3 connection. r=necko-reviewers,valentin
Status: ASSIGNED → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → 95 Branch

Thank you so much for fixing this, and so quickly!

Will this fix wind up in Firefox 91.4 ESR?

Any chance for the fix to be cherry-picked to Firefox 94, or will we have to wait for 95 to be released to stable?

Thanks!

You need to log in before you can comment on or make changes to this bug.