1565701 - Firefox making more HTTP requests than Chrome when loading https://ebay.com

Status: RESOLVED DUPLICATE of bug 1429800

(Core :: Networking, defect, P2)

Description

[Sean Feng [:sefeng]]

Attachment: firefox_chrome_ebay_http_requests.png

The data has been monitored here, some folks may not have access, so I uploaded a screen shot https://app.datadoghq.com/dashboard/vh6-xxd-6ri/httpsebaycom?from_ts=1560374994138&live=true&tile_size=m&to_ts=1562966994138

This is also reproducible by using Web page replay or loading the live site to compare the HAR file, where as Chrome usually took about 150 requests, and we took about 300 requests.

I am going to compare the HAR files to provide more info.

I tried to switch browser.contentblocking.category to standard, strict, custom. None of them changed the result.

Note that, based on my testing, by using Web page replay and Browsertime, this only affects Desktop. GVE makes pretty same amount of requests as Chrome for Android.

Comment 1

[Sean Feng [:sefeng]]

Attachment: chrome.har

upload Chrome Har file

Comment 2

[Sean Feng [:sefeng]]

Attachment: firefox.har

upload Firefox har file

Comment 3

[Sean Feng [:sefeng]]

Attachment: Screenshot from 2019-07-12 19-10-13.png

We loaded much more third party requests by comparing the HAR file. HAR1 is firefox and HAR2 is Chrome

Comment 4

[Honza Bambas (:mayhemer)]

Could you provide diff of urls?

Comment 5

[Sean Feng [:sefeng]]

Mayhemer: if you download the HAR files and use https://compare.sitespeed.io/#domainsHeader to compare them, you can see all the differences. It is very clear.

Comment 6

[Honza Bambas (:mayhemer)]

I personally don't believe this belongs to this component, but we will happily look at this issue first.

Comment 7

[Sean Feng [:sefeng]]

I tried to switch the user agent to Chrome, nothing changed.

Comment 8

[Honza Bambas (:mayhemer)]

Sean, I need to know what exact steps you do to capture the HARs. It's not clear at all from comment 0. Note bug 1468476.

Comment 9

[Honza Bambas (:mayhemer)]

And maybe add changes to preferences or if profiles are clean. Also, in what state is the HTTP cache - clean or primed?

Comment 10

[Honza Bambas (:mayhemer)]

One difference is that we don't accept webp, so we load (equivalent) jpgs instead. Could be bug 1544231.

I can look at the other differences after I have more info on STR.

Comment 11

[Sean Feng [:sefeng]]

Hi Honza, thanks for looking at this.

I used browsertime to collect these HAR files, it was a new tool we added to the mach command, basically it allows us to collect some page load metrics easily. To run it locally, you want to set it up first by running ./mach browsertime --setup, and the exact command I used to collect HAR file was ./mach browsertime -b firefox --video --videoParams.addTimer false --visualMetrics true -n 1 --pageCompleteWaitTime 10000 --firefox.binaryPath '/usr/bin/firefox-nightly' https://ebay.com

This was the command I used to collect Chrome HAR
./mach browsertime -b chrome --video --videoParams.addTimer false --visualMetrics true -n 1 --pageCompleteWaitTime 10000 --chrome.binaryPath '/usr/bin/google-chrome' https://ebay.com

Browsertime would always start a fresh profile for each run, so I believe both Firefox and Chrome was in a clean state.

Comment 12

[Sean Feng [:sefeng]]

By taking Honza's suggestion to use deterministic web pages, I start to think this might be a false alarm.

The HAR file generated by Chrome didn't include everything. For example, by using web page replay, the log showed Chrome had made an request to fetch the mp4 file, 2019/08/01 17:36:20 ServeHTTP(http://www.partridgegetslucky.com/vid/partridgeGetsLucky.mp4): serving 206, 2912257 bytes, but this request didn't get included in the HAR file generated by Chrome.

Since the HAR file wasn't reliable, so I compared the web page replay log again for both Chrome and Firefox loading https://ebay.com. I noticed we made a lot of requests OCSP related requests, for example, something like 2019/08/02 10:11:22 ServeHTTP(http://ocsp.godaddy.com/): serving 200, 5 bytes . After I removed all OCSP related requests, it turned out we made pretty much the same amount of http requests as Chrome.

This might also explain why I didn't see these differences on GVE, probably because we didn't do OCSP on GVE?

Andrew, do you know how datadog collects these data? I think we should confirm OCSP requests are not included in the results.

Comment 13

[Honza Bambas (:mayhemer)]

Thanks for doing this, Sean! Note that OCSP is enabled by default in Firefox. Chrome is not doing OCSP checking. They have their own (chrome-specific) mechanism of certificate invalidation info distribution.

Comment 14

[Andrew Creskey [:acreskey]]

Nice work Sean!
That's an interesting find.

I think the reason you aren't seeing OCSP requests on Android is because (I found out this week) we only do them for a subset of certificates (EV, extended validation)
Look at the pref for android:
https://searchfox.org/mozilla-central/rev/5e660d3dfcba897c8501e3fda1d415565a096e7e/mobile/android/app/mobile.js#421
I've read that these EV certs are now quite rare, and I've only been able to find them on a few sites (e.g. https://github.com/)

So from your results and datadogs I think the har collection is correct and on desktop we do indeed make more requests due to OCSP.
By the way, security folks will replace OCSP with Crlite.

Comment 15

[Honza Bambas (:mayhemer)]

OK, thank you guys! Can we now somehow resolve or conclude this bug? According comment 12, there doesn't seem to be anything else than the (known) difference in OCSP requests.

Comment 16

[Andrew Creskey [:acreskey]]

Maybe close as WONTFIX, since there's nothing to do here? Bug 1429800 will get rid of this request discrepancy.

Comment 17

[Honza Bambas (:mayhemer)]

Definitely not FIXED (no patch).