Websocket cannot send data larger than 500 kilobytes
Categories
(Core :: Networking, defect, P2)
Tracking
()
People
(Reporter: sdaniele3, Assigned: kershaw)
References
(Depends on 1 open bug, Blocks 2 open bugs, Regression)
Details
(Keywords: regression, Whiteboard: [necko-triaged])
Attachments
(6 files)
161.21 KB,
application/zip
|
Details | |
20.00 KB,
text/plain
|
Details | |
16.00 KB,
text/plain
|
Details | |
1.60 MB,
application/zip
|
Details | |
1.11 MB,
application/zip
|
Details | |
48 bytes,
text/x-phabricator-request
|
diannaS
:
approval-mozilla-release+
RyanVM
:
approval-mozilla-esr91+
|
Details | Review |
User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:91.0) Gecko/20100101 Firefox/91.0
Steps to reproduce:
See attached zip file for small sample.
The "websocket.js" file will need to be modified. A server that accepts a websocket request should be configured in the "SERVER_URI" variable. Providing a working websocket server is a little complex and is out of scope for this request. It can be done with various python bits.
Once "websocket.js" has been updated you can serve the folder via python -m http.server
and navigate to localhost.
Click on either of the buttons on the web page in order to create a websocket connection, load some binary data (via some random files I found on my computer), and send it over the socket.
Actual results:
When trying to send a Blob that is over 500 kilobytes Firefox hangs and never sends the data. If the Blob is smaller the data is sent right away.
Expected results:
Data should be sent. WebSocket supports like gigabytes, right?
I tried using mozregression to figure out if this broke at some point, but I ended up with a reggression back in like 2018.. Which is weird and I don't know if it's helpful.
Last Known Good: 2018-11-06
First Known Bad: 2018-11-07
Pushlog: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=6e842238034cd847ede178b4e65ea07704e4ffe6&tochange=5836a60614764631436bf5030c5baa34c676c7a2
mozregression couldn't find builds to get a smaller range. As this was almost three years ago I feel like it may be irrelevant, and I find it surprising nobody has noticed a 500k limit on WebSocket over this time.
Comment 1•2 years ago
|
||
The Bugbug bot thinks this bug should belong to the 'Core::Networking' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.
I am not able to reproduce the issue with Windows 10 + Firefox 91.0.2. This could be limited to Linux.
Assignee | ||
Comment 3•2 years ago
|
||
(In reply to sdaniele3 from comment #2)
I am not able to reproduce the issue with Windows 10 + Firefox 91.0.2. This could be limited to Linux.
I am also not able to reproduce with MacOS + Firefox nightly. Could you try to capture a log? Please add nsWebSocket:5
to MOZ_LOG
env. Thanks.
Here are two captures. One attempting the 400k send (success) and another with the 500k send (failure).
Assignee | ||
Comment 7•2 years ago
•
|
||
The failed log shows that the websocket channel was closed before sending out the data, but I can't find out the reason from the log.
[Parent 155564: Socket Thread]: D/nsWebSocket WebSocketChannel::OnOutputStreamReady: Try to send 583189 of data
[Parent 155564: Socket Thread]: D/nsWebSocket WebSocketChannel::OnOutputStreamReady: write 0 rv 80470007
[Parent 155564: Socket Thread]: D/nsWebSocket WebSocketChannel::OnOutputStreamReady() 7fea14c0b000
[Parent 155564: Socket Thread]: D/nsWebSocket WebSocketChannel::OnOutputStreamReady: Try to send 583189 of data
[Parent 155564: Socket Thread]: D/nsWebSocket WebSocketChannel::OnOutputStreamReady: write 0 rv 80470007
[Parent 155564: Main Thread]: D/nsWebSocket WebSocketChannel::Close() 7fea17021000
Could you try to get the log again with setting MOZ_LOG
to MOZ_LOG=timestamp,rotate:200,nsHttp:5,nsSocketTransport:5,nsWebSocket:5
?
Thanks.
Reporter | ||
Comment 10•2 years ago
|
||
I have attached new logs. With the additional flags I get more files with output so they're attached as zips.
Procedure:
Open firefox in a fresh-ish profile (Was created a few days ago, pretty much for this ticket)
Navigate to localhost:8000, which serves my attached page.zip
Press the button
Wait a few seconds
Close the browser
Assignee | ||
Comment 11•2 years ago
|
||
Apparently, the code here is totally broken. For a temporary workaround, I'd suggest to disable websocket over http2 by turning off network.http.spdy.websockets
.
Assignee | ||
Comment 12•2 years ago
•
|
||
Changing severity to S3, because websocket over http2 is not really common. Users can disable this as a workaround.
Reporter | ||
Comment 13•2 years ago
|
||
Just confirming that network.http.spdy.websockets=false
bypasses the issue.
Assignee | ||
Comment 14•2 years ago
|
||
Updated•2 years ago
|
Assignee | ||
Comment 15•2 years ago
|
||
Since we are close to the end of this cycle, I'll land the patch at the next one.
Updated•2 years ago
|
Assignee | ||
Comment 16•2 years ago
|
||
I am still working on a patch to disable some wpt tests.
Comment 17•2 years ago
|
||
There's a r+ patch which didn't land and no activity in this bug for 2 weeks.
:kershaw, could you have a look please?
For more information, please visit auto_nag documentation.
Assignee | ||
Comment 18•2 years ago
|
||
We'll reconsider the decision about disabling websocket over h2.
Assignee | ||
Comment 19•2 years ago
|
||
Some steps to reproduce this bug locally:
- Adjust the value here to a much smaller value (e.x., 52).
- Run this web-platform test locally by running
./mach test testing/web-platform/tests/websockets/Send-65K-data.any.js --headless
. - Should be able to see the test stuck at
TEST_START: /websockets/Send-65K-data.any.worker.html?wpt_flags=h2
.
Comment 20•1 year ago
|
||
Hello!
I'm on 96.0.3 (64-bit) and experiencing the issue in our app. The app can't send data large/equal than 512Kb over a WebSocket (socket gets stuck). I can trigger the issue calling socket.send(new Uint8Array(512 * 1024))
, it appears as being sent on the network tab, but any subsequent send(...)
calls will not appear there.
Workaround with disabling network.http.spdy.websockets=false
solves the issue for me.
- Kershaw, could you give an update, what's the decision on disabling this flag by default? Should we expect it happening in the coming releases?
- Is there anything I can do to help debugging the issue?
Thanks!
Comment 21•1 year ago
|
||
[Tracking Requested - why for this release]:
impacting a major product
(ask me if you want to know which one)
Assignee | ||
Comment 22•1 year ago
|
||
Does setting network.http.spdy.websockets
or (network.http.http2.websockets
) to false make the product work?
If yes, we might need to consider disabling websocket over http/2.
Comment 23•1 year ago
|
||
comment 20 seems to say exactly that: "Workaround with disabling network.http.spdy.websockets=false solves the issue for me."
Assignee | ||
Comment 24•1 year ago
|
||
(In reply to Randell Jesup [:jesup] (needinfo me) from comment #23)
comment 20 seems to say exactly that: "Workaround with disabling network.http.spdy.websockets=false solves the issue for me."
Right, but I am not sure comment 21 and 20 are about the same thing.
Updated•1 year ago
|
Updated•1 year ago
|
Comment 26•1 year ago
|
||
Dragana, given the webcompat aspect, should we move this to S2?
Comment 27•1 year ago
|
||
Set release status flags based on info from the regressing bug 1434137
Comment 28•1 year ago
|
||
Changing severity to S2 due to the new web-compat aspect of the issue.
Comment 29•1 year ago
|
||
Based on Comment 21 (and the according internal mailing thread), setting this as a WebCompat P1 for now.
Updated•1 year ago
|
Comment 30•1 year ago
|
||
Pushed by kjang@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/75e58c0ca309 Disable websocket over h2, r=necko-reviewers,dragana
![]() |
||
Comment 31•1 year ago
|
||
Backed out for causing http2-websocket.sub.h2.* failures:
https://hg.mozilla.org/integration/autoland/rev/2b820f90b73dabdb37049cea0e4b9b2d94aa49d9
Push which ran failed tasks: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry&revision=567e7a511a30141b4a634ddb5d90f510acfc695e&selectedTaskRun=fOhQWlzdTyClyaKNXNnwmQ.0
Failure log: https://treeherder.mozilla.org/logviewer?job_id=375807591&repo=autoland
TEST-UNEXPECTED-FAIL | /infrastructure/server/http2-websocket.sub.h2.any.worker.html | WSS over h2 - assert_true: expected true got false
TEST-UNEXPECTED-FAIL | /infrastructure/server/http2-websocket.sub.h2.any.html | WSS over h2 - assert_true: expected true got false
Comment 32•1 year ago
|
||
There is also this failure
INFO - TEST-UNEXPECTED-FAIL | /websockets/extended-payload-length.html?wpt_flags=h2 | Application data is 125 byte which means any 'Extended payload length' field isn't used at all. - assert_unreached: close event should not fire Reached unreachable code
Failure log: https://treeherder.mozilla.org/logviewer?job_id=375816829&repo=autoland&lineNumber=19672
Assignee | ||
Comment 33•1 year ago
|
||
Comment 34•1 year ago
|
||
Pushed by kjang@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/eaed92709627 Disable websocket over h2, r=necko-reviewers,dragana
Comment 35•1 year ago
|
||
bugherder |
Assignee | ||
Comment 36•1 year ago
|
||
Note that the patch landed in this bug is just a temporary workaround. In that patch, we disabled websocket over http2
due to this bug.
With this workaround, websocket connections can fallback to use http/1.1 and users should not notice this. This gives us some time to fix this bug properly. Once we have a real fix for this bug, we'll re-enable websocket over http2
.
Updated•1 year ago
|
Comment 37•1 year ago
|
||
Are we planning to uplift this to mozilla-release for 100.0.1 also?
Assignee | ||
Comment 38•1 year ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM] from comment #37)
Are we planning to uplift this to mozilla-release for 100.0.1 also?
We also plan to disable this via Normandy, so I think we don't have to uplift this.
Thanks.
Comment 39•1 year ago
|
||
(In reply to Kershaw Chang [:kershaw] from comment #38)
We also plan to disable this via Normandy, so I think we don't have to uplift this.
What happens if we ship a 100.0.1 release without this change? AFAICT, the Normandy recipe doesn't apply there. Also, waiting for a Normandy rollout isn't as instantaneous as having the pref change set by default from the start.
Assignee | ||
Comment 40•1 year ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM] from comment #39)
(In reply to Kershaw Chang [:kershaw] from comment #38)
We also plan to disable this via Normandy, so I think we don't have to uplift this.
What happens if we ship a 100.0.1 release without this change? AFAICT, the Normandy recipe doesn't apply there. Also, waiting for a Normandy rollout isn't as instantaneous as having the pref change set by default from the start.
This is a good point. Using Normandy here is just for safe - we do 25% rollout first and ramp up to 100% a week later. However, this change has been already on nightly and beta for a while and we didn't get any bug report, so this change should be safe.
I'll request a release uplift. Thanks.
Assignee | ||
Comment 41•1 year ago
|
||
Comment on attachment 9242416 [details]
Bug 1727995 - Disable websocket over h2, r=#necko
Beta/Release Uplift Approval Request
- User impact if declined: Websocket over h2 is not fully functional.
- Is this code covered by automated tests?: Yes
- Has the fix been verified in Nightly?: Yes
- Needs manual test from QE?: No
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): This is already verified on nightly and beta.
- String changes made/needed: N/A
- Is Android affected?: Yes
Comment 42•1 year ago
|
||
Comment on attachment 9242416 [details]
Bug 1727995 - Disable websocket over h2, r=#necko
Approved for 100.0.1
Comment 43•1 year ago
|
||
bugherder uplift |
Updated•1 year ago
|
Comment 44•1 year ago
|
||
I'm not sure how exactly should I verify the fix of this bug. I opened the file attached in a localhost on both Firefox 91 and the latest Firefox Nightly, but I couldn't see any differences between the 2 builds or I'm not looking where I should.
Here's a link comparing the 2 builds on Ubuntu 18.04 x64: https://imgur.com/a/8BV7aLm
I didn't notice any hang on both builds, but I'm not sure where to look if a blob is sent or not.
Could you please help me with some extra details?
Thanks.
Reporter | ||
Comment 45•1 year ago
|
||
The attached file will required some modifications to work as well as a webserver that serves websockets (not provided).
Assignee | ||
Comment 46•1 year ago
|
||
(In reply to Hani Yacoub from comment #44)
I'm not sure how exactly should I verify the fix of this bug. I opened the file attached in a localhost on both Firefox 91 and the latest Firefox Nightly, but I couldn't see any differences between the 2 builds or I'm not looking where I should.
Here's a link comparing the 2 builds on Ubuntu 18.04 x64: https://imgur.com/a/8BV7aLm
I didn't notice any hang on both builds, but I'm not sure where to look if a blob is sent or not.
Could you please help me with some extra details?
Thanks.
Sorry, I can't find a public websocket server for testing this easily.
Assignee | ||
Comment 47•1 year ago
|
||
Comment on attachment 9242416 [details]
Bug 1727995 - Disable websocket over h2, r=#necko
ESR Uplift Approval Request
- If this is not a sec:{high,crit} bug, please state case for ESR consideration: Websocket over h2 is not fully functional.
- User impact if declined: Firefox could not work well for some websites that support websocket over h2.
- Fix Landed on Version: Nightly 101
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): This is already verified on nightly and beta.
Updated•1 year ago
|
Comment 48•1 year ago
|
||
sdaniele3, could you please check if the issue is fixed on your end on Firefox Nighty and Beta?
You can download the builds from here: https://www.mozilla.org/en-US/firefox/channel/desktop/
Thanks.
Comment 49•1 year ago
|
||
Comment on attachment 9242416 [details]
Bug 1727995 - Disable websocket over h2, r=#necko
Approved for 91.10esr.
Comment 50•1 year ago
|
||
bugherder uplift |
Comment 51•1 year ago
|
||
sdaniele3, I'm coming back with the last request here, could you please check if the issue is fixed on Firefox 100.0.1 too so we could close this bug as verified?
You can download the build from here: https://archive.mozilla.org/pub/firefox/candidates/100.0.1-candidates/build1/
Thanks.
Reporter | ||
Comment 52•1 year ago
|
||
I successfully completed the test using "https://archive.mozilla.org/pub/firefox/candidates/100.0.1-candidates/build1/linux-x86_64/en-US/firefox-100.0.1.tar.bz2" and a fresh profile.
Comment 53•1 year ago
|
||
Thank you. I'll change the status of this bug to verified.
Comment 54•1 year ago
|
||
sdaniele3, actually could you please do this last verification on the latest esr build just to make sure that we don't have any issue there also?
https://archive.mozilla.org/pub/firefox/candidates/91.10.0esr-candidates/build1/
Thank you!
Reporter | ||
Comment 55•1 year ago
|
||
I successfully completed the test using "https://archive.mozilla.org/pub/firefox/candidates/91.10.0esr-candidates/build1/linux-x86_64/en-US/firefox-91.10.0esr.tar.bz2" and a fresh profile.
Comment 56•1 year ago
|
||
Thank you!
Updating the tracking flag accordingly.
Assignee | ||
Comment 57•6 months ago
|
||
(In reply to Anton Fisher from comment #20)
Hello!
I'm on 96.0.3 (64-bit) and experiencing the issue in our app. The app can't send data large/equal than 512Kb over a WebSocket (socket gets stuck). I can trigger the issue calling
socket.send(new Uint8Array(512 * 1024))
, it appears as being sent on the network tab, but any subsequentsend(...)
calls will not appear there.Workaround with disabling
network.http.spdy.websockets=false
solves the issue for me.
- Kershaw, could you give an update, what's the decision on disabling this flag by default? Should we expect it happening in the coming releases?
- Is there anything I can do to help debugging the issue?
Thanks!
Hi Anton,
We've improved our code to support WebSocket over Http/2 recently.
Could you help us verify if this issue is fixed by enabling this pref network.http.http2.websockets
at your side?
Thanks.
Assignee | ||
Comment 58•6 months ago
|
||
The reporter already replied via email.
Description
•