1852924 - H3 upload Speed still slow. Three times slower than other browsers.

Status: ASSIGNED

(Core :: Networking: HTTP, defect, P1)

Description

[Andre]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36

Steps to reproduce:

The upload speed of Firefox is still very slow compared to other browsers out there.

There was an older bug about this, that has been marked as fixed. But I don't see why it has been fixed as the issue still persists: https://bugzilla.mozilla.org/show_bug.cgi?id=1596576

Actual results:

Uploading a file to YouTube for example:

Filesize: 8 GB
Upload takes 8 Minutes to upload via Edge or Chrome.
Upload takes 28 Minutes to upload with Firefox (118 beta 8 and 117).

Tested with upload capacity of 200 Mbit/s. The upload in Chrome and Edge is using the whole capacity for the upload. It is not possible to go faster than 8 minutes. Why does Firefox take so long?

Expected results:

Upload speed of Firefox should not be slower than Edge / Chrome.

Comment 1

[Manuel Bucher [:manuel]]

Hi, this is likely slow H3 upload speed (Bug 1708868).

We do experiment with a fix (Bug 1851908) and would appreciate help in testing it.

• Make sure you see the preference network.http.http3.cc_algorithm (Firefox Nightly 119 and onwards) https://www.mozilla.org/en-US/firefox/119.0a1/releasenotes/
• Measure the upload throughput while on the default setting (1 for Cubic).
• Modify this preference to 0 (indicating newReno).
• Restart Firefox and measure the upload throughput again.

Comment 2

[Andre]

(In reply to Manuel Bucher [:manuel] from comment #1)

Hi, this is likely slow H3 upload speed (Bug 1708868).

We do experiment with a fix (Bug 1851908) and would appreciate help in testing it.

• Make sure you see the preference network.http.http3.cc_algorithm (Firefox Nightly 119 and onwards) https://www.mozilla.org/en-US/firefox/119.0a1/releasenotes/
• Measure the upload throughput while on the default setting (1 for Cubic).
• Modify this preference to 0 (indicating newReno).
• Restart Firefox and measure the upload throughput again.

I have downloaded the 119.0a1 nightly build and uploaded with the default setting for network.http.http3.cc_algorith = 1
Upload time for a 5.8 GB file to YouTube estimating at 36 minutes (Same file would be about 6 minutes on Chrome and Edge).

After setting network.http.http3.cc_algorithm = 0 as suggested and restarting the nightly firefox build I uploaded the same file again which was then estimated at 34 minutes... which would mean no change in upload speed at all on my machine.

Comment 3

[Manuel Bucher [:manuel]]

Thanks for trying it out so fast. This answer makes this report more interesting. Knowing whether this is h2 or h3 would be good. Can you look into about:networking which http version is used for youtube.com? You can also try to disable h3 by setting network.http.http3.enable to false and see if the upload speed is still slow.

If it is h3, we can probably track it in the other bug. If it is h2, we would need a new one.

Comment 4

[Manuel Bucher [:manuel]]

(undoing the accidental title change)

Comment 5

[Andre]

HTTP/3 is used on upload.youtube.com according to about:networking.

Comment 6

[Andre]

It's also worth noting that the initial estimate of about 30 minutes for uploading that file is steadily going up while uploading. This makes Firefox not three but up to 8 times slower that other browsers.

Comment 7

[Manuel Bucher [:manuel]]

Okay, lets keep track of this. Andrew is currently working on upload speed performance and will get back if necessary.

Comment 8

[taylor]

I posted on the other bug thread a while ago, but just also posting here that I have the exact same issue. Also I just built a new PC last week, exact same problem - it exists on every Windows machine. I'm on beta not nightly so don't have 119 yet but I'll try the 'cc_algorithm' change when I get 119, but sounds like it may not make a difference based on the above.

But yeah, anytime I have to upload something, I switch to chrome or edge.

Comment 9

[Andrew Creskey [:acreskey]]

Yes, please do try the network.http.http3.cc_algorithm pref when you have 119.
(Be sure to exit and restart Firefox to ensure that a new connection is made).
We're seeing that it can make a large difference in our tests, which we are now expanding to a nightly experiment.
But it all depends on your network characteristics, so your findings are valuable.

Comment 10

[Andre]

(In reply to Andrew Creskey [:acreskey] from comment #9)

Yes, please do try the network.http.http3.cc_algorithm pref when you have 119.
(Be sure to exit and restart Firefox to ensure that a new connection is made).
We're seeing that it can make a large difference in our tests, which we are now expanding to a nightly experiment.
But it all depends on your network characteristics, so your findings are valuable.

As I stated earlier, I have tested that. It does change nothing unfortunately.

Comment 11

[Bas Schouten (:bas.schouten)]

I can trivially reproduce this locally on two different Windows machines. 1GB fiber connection. 4-5x speed difference for me. (50seconds vs 4-5minutes for a 4GB video)

The Performance Impact Calculator has determined this bug's performance impact to be high. If you'd like to request re-triage, you can reset the Performance Impact flag to "?" or needinfo the triage sheriff.

Platforms: Windows
Impact on site: Causes noticeable jank
Websites affected: Major
[x] Able to reproduce locally
[x] Multiple reporters

I agree with the triage calculator marking this as High. When uploading large videos this is a significant disruption to your workflow and the type of thing that will make people feel forced to use Chrome.

Comment 12

[BugBot [:suhaib / :marco/ :calixte]]

The severity field for this bug is set to S3. However, the Performance Impact field flags this bug as having a high impact on the performance.
:edgul, could you consider increasing the severity of this performance-impacting bug? Alternatively, if you think the performance impact is lower than previously assessed, could you request a re-triage from the performance team by setting the Performance Impact flag to ??

For more information, please visit BugBot documentation.

Comment 13

[edgul]

I'm going to raise this to S2 since it seems like adjusting the cc doesn't change the issue, so it may not be related to Bug 1848840. Noting also that, as mentioned elsewhere in this bug, H3 upload speed work is ongoing

Comment 14

[Kershaw Chang [:kershaw]]

Attachment: Bug 1852924 - Fill the stream's buffer with data before calling ProcessOutput, r=#necko

Note that this patch only improves the upload performance a bit.
We should ensure the stream's buffer is maximally filled before calling ProcessOutput to create packets.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/232401381295
Fill the stream's buffer with data before calling ProcessOutput, r=necko-reviewers,jesup

Comment 16

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/232401381295

Comment 17

[Kershaw Chang [:kershaw]]

Attachment: Bug 1852924 - When pacer returns a small delay, pretend to wait and get the next packet to send, r=mt

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f5d7c3a00f2b
When pacer returns a small delay, pretend to wait and get the next packet to send, r=mt

Comment 19

[Cristina Horotan [:chorotan]]

Backed out changeset f5d7c3a00f2b (Bug 1852924) for causing dt failures at mod.rs

Backout: https://hg.mozilla.org/integration/autoland/rev/692f8575939392967f223a9f7514568acc6a2ac7

Failure push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=linux%2C18.04%2Cx64%2Cwebrender%2Cdebug%2Cmochitests%2Cwith%2Chttp%2F3%2Cserver%2Ctest-linux1804-64-qr%2Fdebug-mochitest-devtools-chrome-http3%2Cdt13&revision=f5d7c3a00f2bdc43070a0a547ddcd9027f3a2d02&selectedTaskRun=RDldV3CDRs6WiAlsGuWdaw.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=435181156&repo=autoland&lineNumber=9239

Comment 20

[Manuel Bucher [:manuel]]

For anyone interested: The main problems remaining are these two issues tracked in neqo.

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/75ebf5892d17
When pacer returns a small delay, pretend to wait and get the next packet to send, r=mt

Comment 22

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/75ebf5892d17

Comment 23

[Noemi Erli[:noemi_erli]]

Backed out changeset 75ebf5892d17 (Bug 1852924) for frequent fail rate in Bug 1751116

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=Rl1_KoIBTYCwEU__iWnbYw.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=75ebf5892d17700c1be42f282ec32b8b1d696f49

Log: https://treeherder.mozilla.org/logviewer?job_id=435577491&repo=autoland&lineNumber=8951

Backout: https://hg.mozilla.org/integration/autoland/rev/b97795caaddfe442e4e17091b9437a0f216f1622

Comment 24

[Cosmin Sabou [:CosminS]]

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/b97795caaddfe442e4e17091b9437a0f216f1622

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e85c337798a4
When pacer returns a small delay, pretend to wait and get the next packet to send, r=mt

Comment 26

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/e85c337798a4

Comment 27

[Kershaw Chang [:kershaw]]

We have landed some patches recently to improve the HTTP/3 upload performance.
Could you try to use the latest nightly version and test again?
If the upload speed is still slow, could you try to record a log by visiting about:logging and selecting the logging preset to HTTP/3 upload speed and Logging to a file?

Thanks.

Comment 28

[Andre]

Tested with 122.0a1 (2023-11-29) as requested.

The result looks very promising. A 4.8 GB File was estimated at 7 Minutes (slightly higher than Google or Edge, who both estimated ~ 5 MInutes).
The upload then took ~ 6 Minutes, which is a huge improvement and very close to Google and Edge who took the estimated 5 Minutes.

I would say Firefox, with these updates, is almost on par with the other browsers, which makes it usable again for me. Thank you very much!

Comment 29

[Bas Schouten (:bas.schouten)]

I see similar results. Not on my normal fast internet as I'm traveling. But I was able to upload a 2.8GB file in 1 minute 15 seconds in Firefox vs 1 minute 5 seconds in Google Chrome. Still a difference, but nowhere near as disabling as it was before. I would consider this issue as it stands resolved.

My recommendation would be to resolve it and remove the leave-open keyword.

Andre, thank you for your active participation in helping resolve this issue!

Comment 30

[taylor]

I can give this a test on my PC as well a bit later tonight! I have 1000 Fiber and I can test it with a large terabyte upload and let you all know the results compared to chrome/edge!

Comment 31

[taylor]

I'm not seeing great results on my end. I ended up not having access to a large TB file, but I did upload a 72GB file and here are the results:

Edge: 27 Minutes
FireFox Nightly 122.0a1: 51 minutes

So Firefox is taking almost twice as long. Definitely would not consider this resolved Bas.

Comment 32

[taylor]

PS - for anyone testing, I would suggest testing with a somewhat large file. Small files (like a couple gb) are harder to tell the difference when they only take a couple minutes. With larger files, the speed/time differences are much more apparent.

Comment 33

[Animesh Bhattacharya [:AnimeshB]]

I can attest to taylor's experience. I'm on a 300 mbps connection and the difference is still massive even with a 550 megabyte file, though significantly improved from before:

~550 megabyte file upload to Google Drive:
Edge: 21 seconds
Firefox Nightly 122.0a1: 42 seconds
Firefox 120.0.1: 2 minutes and 43 seconds

Nightly is taking at least twice as long on my PC to upload the same file. The actual upload speeds with Firefox (According to Windows task manager) can start off with nearly ~200 mbps, but that quickly plummets to 100-120 mbps and stays there for most of the upload. Edge reaches 300 mbps instantly and stays there for for the majority of the upload.

Comment 34

[Kershaw Chang [:kershaw]]

(In reply to TexlGyTrail from comment #33)

I can attest to taylor's experience. I'm on a 300 mbps connection and the difference is still massive even with a 550 megabyte file, though significantly improved from before:

~550 megabyte file upload to Google Drive:
Edge: 21 seconds
Firefox Nightly 122.0a1: 42 seconds
Firefox 120.0.1: 2 minutes and 43 seconds

Nightly is taking at least twice as long on my PC to upload the same file. The actual upload speeds with Firefox (According to Windows task manager) can start off with nearly ~200 mbps, but that quickly plummets to 100-120 mbps and stays there for most of the upload. Edge reaches 300 mbps instantly and stays there for for the majority of the upload.

Thanks for testing.
Could you use Nightly 122.0a1 and follow the steps in comment #27 to record a log? It would be very useful to analyze the problem with a log.

Comment 35

[Animesh Bhattacharya [:AnimeshB]]

Thanks for testing.
Could you use Nightly 122.0a1 and follow the steps in comment #27 to record a log? It would be very useful to analyze the problem with a log.

Here's the moz_log file from uploading that same file to Google drive like before:

https://drive.google.com/file/d/1gbclOYIuffu4IG1IVjIrbEtLdRX4B5Hy/view?usp=sharing

Alternative link just in case:
https://mega.nz/file/tWsW2QSB#6LQgrPHJR6erh8bkLWONVwGn_ItgQc90Xp5FYKeiK08

Comment 36

[Manuel Bucher [:manuel]]

Attachment: Firefox_Nightly_122.0a1.moz_log.png

Thanks for the log. It is really helpful to see the networking condition. This is what I gathered using my log visualizer.

1. In this network log all detected packet losses were really lost packets. No spurious retransmission were recorded, so in this network condition RACK wouldn't improve upload speed.
2. We do see a time frame of 300ms without sending packets. That is tracked in neqo #1474.
3. The second drop in packets is more interesting, because it more than 1s and it fails to recover bytes_in_flight back to the congestion window, but might still be the same bug (neqo #1474). This might be due to other websites/connections affecting h3 upload.
4. We don't send packets fast enough, that is tracked in neqo #1483, but neqo #1487 could also be a positive impact with not too much effort.

I think focusing on neqo #1487 first is good, because it is actionable. We saw that in other networking conditions RACK is really helpful, so I would tackle that afterwards. And the other two issues should also be looked at :).

Comment 37

[Greg Hess]

Moving this bug out of priority queue, remaining work to close the gap with Chrome will be coordinated via project plan.