1934982 - Fenix downloads over HTTP/1.1 often 20-30% slower than Chrome

Status: NEW

(Core :: Networking: HTTP, defect, P2)

Description

[Marc Leclair [:mleclair]]

Steps to reproduce

Download any files from testfile.org. I used this script I made to test my download speed on firefox vs chrome (still needs some work for the links and browsers supported)

Expected behavior

It should be the same, or almost the same

Actual behavior

On my local wifi (both browsers showed ~ 800 up and down before the test), I get the following:
For a 250MB file:

Download speed comparison:
Firefox: 9.176 seconds
Chrome: 8.174 seconds

For a 500MB file:

Download speed comparison:
Firefox: 26.878 seconds
Chrome: 16.465 seconds

For a 1GB file:

Download speed comparison:
Firefox: 86.982 seconds
Chrome: 41.364 seconds

for the last one, I have this profile: https://share.firefox.dev/4eYZlbt for the slow download. Not sure if it was helpful

Device information

• Firefox version: 132
• Android device model: pixel 9 pro fold
• Android OS version:14

Any additional information?

Comment 1

[Andrew Creskey [:acreskey]]

Good bug Marc.
As we were discussing, can you find out which version of the HTTP protocol is being used for the downloads? Likely HTTP/2 or HTTP/3.
The easiest way is probably with about:debugging and using dev tools (network tab) to look at the download requests.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

GCM_DecryptAEAD() is taking 22% of the time (and a good chunk of that 22% in CTR_Update, gcm_HashMult_hw, and _memcmp_aarch64 - not sure why nss isn't using the secure memcmp - the source for GCM_DecryptAEAD() calls NSS_SecureMemcmp(), but not memcmp())

Comment 3

[John Schanck [:jschanck] (out of office Aug 11-18)]

I think we have a good idea of what's going on in Bug 1935190. I'm not sure that Bug 1935190 explains the full performance difference that we're seeing here, so I think it makes sense to keep this bug open.

not sure why nss isn't using the secure memcmp

These memcmps are for counter rollover detection. The number of times that the counter is incremented can be inferred from public data, so there's no need for a constant time comparison.

Comment 4

[Andrew Creskey [:acreskey]]

Marc, it sounds like you're on a faster network, 800 Mbps up and down, correct?

If you have any ability to throttle your connection, can you also try on a slower network? (e.g. RogersInABox, or throttling via a host machine, like Apple's Link Conditioner)

On my 30Mpbs home wifi connection, 28ms latency, I'm seeing Fenix perform comparably to Chrome.

500 MB file
Download speed comparison:
Fenix Nightly: 91 seconds
Chrome Release: 98 seconds

But it sounds like we're CPU bound on faster networks.

Comment 5

[Andrew Creskey [:acreskey]]

On the 500 MB file I'm seeing it redirect to https://testfileorg.netwet.net/500MB-CZIPtestfile.org.zip and download this one over HTTP/1.1

Comment 6

[Randell Jesup [:jesup] (needinfo me)]

From the profile, the only networking-related thing taking CPU time is CTR_Update/GCM_DecryptAEAD. This is being fixed in the 136 cycle in bug 1936680. Mark, please retest once the NSS fix lands, since I don't think this will make a major change. The other things I see taking time are largely things like JNI interface code, which isn't networking

Comment 7

[John Schanck [:jschanck] (out of office Aug 11-18)]

The NSS change is in the latest Nightly.

Comment 8

[Marc Leclair [:mleclair]]

After the changes, I just ran nightly on 136.0a1 and still get major differences. On 1.3Gb downloads, I get ~20-70 seconds differences (it does vary) and 5Gb pushes the 200 seconds and beyond difference. I will try to gather a profile a bit later.

Comment 9

[Marc Leclair [:mleclair]]

this is the profile I got with my current nightly downloading a 5gb file (took around 179.63 seconds according to the logs). This is my profile with Chrome which took around 67 seconds

Comment 10

[Andrew Creskey [:acreskey]]

(In reply to Marc Leclair [:mleclair] from comment #9)

this is the profile I got with my current nightly downloading a 5gb file (took around 179.63 seconds according to the logs). This is my profile with Chrome which took around 67 seconds

Thank you Marc -- can I also ask you for a gecko profile of Fenix downloading the 5gb file?

Comment 11

[Marc Leclair [:mleclair]]

Here's a gecko profile of the download. This one took ~ 161 seconds. I was in another room when I did the tests which may be why they were a tad higher. In the same room where I ran the above profile ( phone hadn't moved), chrome took ~ 73 seconds

For more info: Firefox seems to get worse when the connection worsens, but not chrome. The download that took around 200 seconds was in a spot where the connection was 300 down / up. (chrome took 67 seconds). The download above that took 161 seconds was in a room with 1gb down / up and chrome stayed relatively the same.

Comment 12

[Andrew Creskey [:acreskey]]

(In reply to Marc Leclair [:mleclair] from comment #11)

Here's a gecko profile of the download. This one took ~ 161 seconds. I was in another room when I did the tests which may be why they were a tad higher. In the same room where I ran the above profile ( phone hadn't moved), chrome took ~ 73 seconds

For more info: Firefox seems to get worse when the connection worsens, but not chrome. The download that took around 200 seconds was in a spot where the connection was 300 down / up. (chrome took 67 seconds). The download above that took 161 seconds was in a room with 1gb down / up and chrome stayed relatively the same.

Thanks Marc, but for some reason this profile is missing the key elements including the network requests (other than the telemetry updates) and the content process.

Can you try again - maybe a smaller file if need and start collecting before the request is made?

Comment 13

[Randell Jesup [:jesup] (needinfo me)]

Andrew didn't NI mark. Can we get some more profiles? Thanks for doing this testing!

Comment 14

[Valentin Gosu [:valentin] (he/him)]

Hi Marc, any updates here? Thanks!

Comment 15

[Marc Leclair [:mleclair]]

So here's what I got . This is a google sheet with different tests on different connections.

TL;DR: Downloads seem better but where chrome is constant, we'll sometimes be much worse. However, this seems to be more obvious on larger files downloaded.

Comment 16

[Andrew Creskey [:acreskey]]

(In reply to Marc Leclair [:mleclair] from comment #15)

So here's what I got . This is a google sheet with different tests on different connections.

TL;DR: Downloads seem better but where chrome is constant, we'll sometimes be much worse. However, this seems to be more obvious on larger files downloaded.

@mleclair, for a given row are the values ("Download Speed (seconds)"and "Speed Test (Download Mbps)" ) reported from the site or manually measured and calculated via the browser UI?

e.g. test 4
Chrome and Firefox both take about 210 seconds to download the file, 210.92 and 210.385, but the reported values for Speed Test (Download Mbps) are quite different, 200.5 Mbps for Chrome and 256.55 Mbps for FIrefox (Firefox roughly 20% faster).

Some observations:
• I don't see any scenarios where downloads in Fenix take twice as long as Chrome (but sometimes it's 20% slower, sometimes faster)
• From looking at the profiles, all downloads were made over HTTP/1.1 in Firefox

Comment 17

[Marc Leclair [:mleclair]]

1. Download Speed (Seconds) was done through the script I linked above. As for the speed, I ran it manually with the google speed test (and double checked it with Ookla speedtest as well)

2. Yes, that's right, it isn't 2x as worse anymore. As I mentioned, the behavior I now see is sometimes 20% worse but I don't see the results I used to see when the bug was first opened. It is worth noting that my Release version has been updated compared to when I first noticed the issue. The 20% worse case usually happens 1 in 4-5 runs, and usually mostly noticeable on bigger download

3. Is http3 enabled by default on release? On my nightly, it looks like its enabled. I couldn't find a website that had downloads through http3.

Comment 18

[Andrew Creskey [:acreskey]]

(In reply to Marc Leclair [:mleclair] from comment #17)

1. Download Speed (Seconds) was done through the script I linked above. As for the speed, I ran it manually with the google speed test (and double checked it with Ookla speedtest as well)

2. Yes, that's right, it isn't 2x as worse anymore. As I mentioned, the behavior I now see is sometimes 20% worse but I don't see the results I used to see when the bug was first opened. It is worth noting that my Release version has been updated compared to when I first noticed the issue. The 20% worse case usually happens 1 in 4-5 runs, and usually mostly noticeable on bigger download

Thanks Marc.

Given what you're seeing, I would consider renaming the bug, "Fenix downloads over HTTP/1.1 often 20-30% slower than Chrome" or something similar. (Although we seem to be roughly comparable via Google Speed Test and Ookla?)

It would also be helpful if you were able to test out downloads over HTTP/2 or HTTP/3

3. Is http3 enabled by default on release? On my nightly, it looks like its enabled. I couldn't find a website that had downloads through http3.

HTTP/3 is enabled on release, but it requires the server to support it and Firefox and the server to agree on the protocol version for that connection.
Generally this will happen on a subsequent visit to the site.

I put this test file on a public google drive, it should be accessible via HTTP/3 on reconnect.
https://drive.google.com/file/d/1NXG0ZjVZCXqogZINsY_8AVpySlongYAM/view?usp=drive_link

But I'm not sure if your test script will work well with google drive.
If it doesn't, let me know and I can setup a different HTTP/3 server for testing.

Comment 19

[Kershaw Chang [:kershaw]]

According to this test results, when downloading a single 32MB file over HTTP/2, Firefox is only slightly slower than Chrome.
For example, in the 300M_40ms case, Firefox achieves a goodput of 244.59 Mbps ± 1.27%, while Chrome reaches 251.14 Mbps ± 1.27%.

However, the result in comment #15 suggests that the observed slowness occurs with HTTP/1.1, which is currently not supported by our benchmarking framework.

I’ll investigate this bug as part of the project to expand our benchmark framework. The first step will be to add support for HTTP/1.1 and Android.