Compare Firefox pageload performance relative to Chrome in live sites and mitmproxy playback
Categories
(Core :: Performance: General, task, P3)
Tracking
()
People
(Reporter: acreskey, Assigned: acreskey)
References
Details
Attachments
(2 files)
tp6 cold 1-10, live_and_recorded, MacBook pro
We currently compare Firefox pageload performance to Chrome using mitmproxy recordings.
e.g.
https://health.graphics/quantum/tp6?test=cold-loadtime&platform=win64&past=month&ending=2019-10-23
This allows us to use static content and gives a reasonable degree of reproducibility.
However it is known that measuring pageload performance by playback through a proxy bypasses or modifies numerous performance-sensitive systems including but not limited to:
-DNS resolution
-TCP connection establishment
-TLS connection (including cert chain verification, OCSP on desktop, cipher-suite selection)
-HTTP/2 coalescing and prioritization
-Network caching
-Speculative loading
-Etc
The purpose of this bug is to collect results and compare Firefox pageload performance relative to Chrome under both live and mitmproxy environments.
We should then be able to determine if our relative performance under mitmproxy playback provides a meaningful measure of our relative performance in an end user environment: live sites.
|
Assignee | |
Comment 1•6 years ago
|
||
I've run a comparison of the first 10 tp6 cold page load tests on MacBook Pro [attached distribution]
|
|
Assignee | |
Comment 2•6 years ago
|
||
I don't consider any of these results to be conclusive, they are just data points.
But this is what I'm seeing.
First a summary of these results:
> summary(subset(df, mode=="firefox_live"))
mode test fcp loadtime
chrome_live : 0 Length:500 Min. : 463.0 Min. : 824
firefox_live :500 Class :character 1st Qu.: 715.8 1st Qu.: 1820
chrome_mitmproxy : 0 Mode :character Median : 943.5 Median : 2274
firefox_mitmproxy: 0 Mean :1143.8 Mean : 2771
3rd Qu.:1320.0 3rd Qu.: 3507
Max. :9854.0 Max. :21038
> summary(subset(df, mode=="chrome_live"))
mode test fcp loadtime
chrome_live :500 Length:500 Min. : 121.4 Min. : 507
firefox_live : 0 Class :character 1st Qu.: 385.9 1st Qu.:1499
chrome_mitmproxy : 0 Mode :character Median : 611.0 Median :1991
firefox_mitmproxy: 0 Mean : 627.6 Mean :2041
3rd Qu.: 798.9 3rd Qu.:2434
Max. :2152.4 Max. :9729
>
> summary(subset(df, mode=="firefox_mitmproxy"))
mode test fcp loadtime
chrome_live : 0 Length:500 Min. : 306.0 Min. : 379
firefox_live : 0 Class :character 1st Qu.: 589.2 1st Qu.:1385
chrome_mitmproxy : 0 Mode :character Median : 689.0 Median :1607
firefox_mitmproxy:500 Mean : 770.4 Mean :1511
3rd Qu.: 933.2 3rd Qu.:1713
Max. :1417.0 Max. :2368
> summary(subset(df, mode=="chrome_mitmproxy"))
mode test fcp loadtime
chrome_live : 0 Length:500 Min. :119.5 Min. : 466
firefox_live : 0 Class :character 1st Qu.:291.3 1st Qu.: 875
chrome_mitmproxy :500 Mode :character Median :326.9 Median : 975
firefox_mitmproxy: 0 Mean :313.0 Mean :1107
3rd Qu.:362.5 3rd Qu.:1183
Max. :671.9 Max. :2971
|
|
Assignee | |
Comment 3•6 years ago
•
|
||
Just looking at the medians, we have:
Live sites: Chrome ~12% faster in the median:
Firefox live median: 2274
Chrome live median: 1991
Mitmproxy playback: Chrome ~40% faster in the median:
Firefox mitmproxy median: 1607
Chrome mitmproxy median: 975
|
|
Assignee | |
Comment 4•6 years ago
|
||
If I take a summary of the first site, tp6-amazon-cold, the relative performance is quite different, live to mitmproxy.
> summary(subset(df, mode=="firefox_live" & test=="raptor-tp6-amazon-cold"))
mode test fcp loadtime
chrome_live : 0 Length:50 Min. : 744.0 Min. :1289
firefox_live :50 Class :character 1st Qu.: 935.5 1st Qu.:1510
chrome_mitmproxy : 0 Mode :character Median : 994.5 Median :1699
firefox_mitmproxy: 0 Mean :1213.0 Mean :1947
3rd Qu.:1085.0 3rd Qu.:1859
Max. :9854.0 Max. :9973
> summary(subset(df, mode=="chrome_live" & test=="raptor-tp6-amazon-cold"))
mode test fcp loadtime
chrome_live :50 Length:50 Min. : 603.2 Min. :1812
firefox_live : 0 Class :character 1st Qu.: 688.8 1st Qu.:2181
chrome_mitmproxy : 0 Mode :character Median : 758.5 Median :2266
firefox_mitmproxy: 0 Mean : 799.6 Mean :2271
3rd Qu.: 893.0 3rd Qu.:2371
Max. :1187.1 Max. :3207
> summary(subset(df, mode=="firefox_mitmproxy" & test=="raptor-tp6-amazon-cold"))
mode test fcp loadtime
chrome_live : 0 Length:50 Min. :640.0 Min. :1070
firefox_live : 0 Class :character 1st Qu.:674.5 1st Qu.:1342
chrome_mitmproxy : 0 Mode :character Median :692.5 Median :1378
firefox_mitmproxy:50 Mean :701.9 Mean :1383
3rd Qu.:734.8 3rd Qu.:1431
Max. :814.0 Max. :1630
> summary(subset(df, mode=="chrome_mitmproxy" & test=="raptor-tp6-amazon-cold"))
mode test fcp loadtime
chrome_live : 0 Length:50 Min. :350.9 Min. :1316
firefox_live : 0 Class :character 1st Qu.:359.9 1st Qu.:1339
chrome_mitmproxy :50 Mode :character Median :364.5 Median :1358
firefox_mitmproxy: 0 Mean :371.6 Mean :1382
3rd Qu.:368.7 3rd Qu.:1408
Max. :570.9 Max. :1662
Live sites: Firefox ~33% faster in the median:
Firefox live median: 1699
Chrome live median: 2266
Mitmproxy playback: Chrome ~1% faster in the median:
Firefox mitmproxy median: 1378
Chrome mitmproxy median: 1358
|
|
Assignee | |
Updated•6 years ago
|
|
|
Assignee | |
Comment 5•6 years ago
|
||
I have some long-running tests going on the Dell Precision (Windows10).
|
|
Assignee | |
Comment 6•6 years ago
|
||
This a plot of onload event timing for the same tp6 1-10 cold load tests on the high end Dell Precision, Windows 10.
|
|
Assignee | |
Comment 7•6 years ago
|
||
From a quick look at the summary, I'm seeing that these results also differ, live to mitmproxy playback:
> summary(subset(df, mode=="firefox_live"))
mode test fcp loadtime
chrome_live : 0 Length:500 Min. : 437.0 Min. : 946
firefox_live :500 Class :character 1st Qu.: 831.2 1st Qu.: 1970
chrome_mitmproxy : 0 Mode :character Median :1007.0 Median : 2506
firefox_mitmproxy: 0 Mean :1168.7 Mean : 2783
3rd Qu.:1560.2 3rd Qu.: 3007
Max. :4270.0 Max. :15841
> summary(subset(df, mode=="chrome_live"))
mode test fcp loadtime
chrome_live :500 Length:500 Min. : 102.0 Min. : 649
firefox_live : 0 Class :character 1st Qu.: 450.2 1st Qu.:1500
chrome_mitmproxy : 0 Mode :character Median : 668.8 Median :1848
firefox_mitmproxy: 0 Mean : 700.1 Mean :1979
3rd Qu.: 851.9 3rd Qu.:2288
Max. :2850.8 Max. :8408
>
> summary(subset(df, mode=="firefox_mitmproxy"))
mode test fcp loadtime
chrome_live : 0 Length:500 Min. : 449 Min. :1540
firefox_live : 0 Class :character 1st Qu.:1020 1st Qu.:2120
chrome_mitmproxy : 0 Mode :character Median :1198 Median :2478
firefox_mitmproxy:500 Mean :1233 Mean :2589
3rd Qu.:1468 3rd Qu.:2826
Max. :2114 Max. :4146
> summary(subset(df, mode=="chrome_mitmproxy"))
mode test fcp loadtime
chrome_live : 0 Length:500 Min. : 114.1 Min. : 550.0
firefox_live : 0 Class :character 1st Qu.: 405.1 1st Qu.: 915.8
chrome_mitmproxy :500 Mode :character Median : 506.9 Median : 1108.5
firefox_mitmproxy: 0 Mean : 492.8 Mean : 1348.2
3rd Qu.: 583.5 3rd Qu.: 1433.0
Max. :1324.1 Max. :10546.0
Live sites: Chrome 26% faster in the median:
Firefox live median: 2506
Chrome live median: 1848
Mitmproxy playback: Chrome 55% faster in the median:
Firefox mitmproxy median: 2478
Chrome mitmproxy median: 1108.5
|
|
Assignee | |
Comment 8•6 years ago
|
||
I won't compare performance on amazon because it turns out that amazon's mitmproxy recording is missing some resources.
But this is the summary of the next one, raptor-tp6-docs-cold, which is a google docs page:
> summary(subset(df, mode=="firefox_live" & test=="raptor-tp6-docs-cold"))
mode test fcp loadtime
chrome_live : 0 Length:50 Min. : 998 Min. :2163
firefox_live :50 Class :character 1st Qu.:1624 1st Qu.:2402
chrome_mitmproxy : 0 Mode :character Median :1718 Median :2472
firefox_mitmproxy: 0 Mean :1602 Mean :2473
3rd Qu.:1771 3rd Qu.:2545
Max. :1911 Max. :2795
> summary(subset(df, mode=="chrome_live" & test=="raptor-tp6-docs-cold"))
mode test fcp loadtime
chrome_live :50 Length:50 Min. : 754.8 Min. :1725
firefox_live : 0 Class :character 1st Qu.: 858.3 1st Qu.:1996
chrome_mitmproxy : 0 Mode :character Median : 894.4 Median :2044
firefox_mitmproxy: 0 Mean : 909.6 Mean :2496
3rd Qu.: 932.7 3rd Qu.:3289
Max. :1165.8 Max. :8408
> summary(subset(df, mode=="firefox_mitmproxy" & test=="raptor-tp6-docs-cold"))
mode test fcp loadtime
chrome_live : 0 Length:50 Min. :1235 Min. :1834
firefox_live : 0 Class :character 1st Qu.:1292 1st Qu.:1876
chrome_mitmproxy : 0 Mode :character Median :1313 Median :1919
firefox_mitmproxy:50 Mean :1311 Mean :1928
3rd Qu.:1322 3rd Qu.:1964
Max. :1410 Max. :2136
> summary(subset(df, mode=="chrome_mitmproxy" & test=="raptor-tp6-docs-cold"))
mode test fcp loadtime
chrome_live : 0 Length:50 Min. :385.0 Min. : 1835
firefox_live : 0 Class :character 1st Qu.:405.3 1st Qu.: 2093
chrome_mitmproxy :50 Mode :character Median :410.0 Median : 2137
firefox_mitmproxy: 0 Mean :415.9 Mean : 2752
3rd Qu.:418.2 3rd Qu.: 2214
Max. :584.7 Max. :10546
Here I'm seeing another result in which the relative performance between live and mitmproxy playback is inconsistent:
Live tp6-docs: Chrome 17% faster in the median:
Firefox live median: 2472
Chrome live median: 2044
Mitmproxy playback: Firefox 11% faster in the median:
Firefox mitmproxy median: 1919
Chrome mitmproxy median: 2137
|
||
Updated•6 years ago
|
|
|
Assignee | |
Comment 9•6 years ago
|
||
Resolving:
From the data I collected comparing Firefox pageload performance to Chrome via mitmproxy playback is not ideal because the results do not translate to live site performance.
Description
•