1583220 - Visual metrics: Fenix is significantly slower than Chrome for the site https://cnn.com

Status: NEW

(Core :: Performance, defect, P2)

Description

[Andrew Creskey [:acreskey]]

Fenix is significantly slower than Chrome 76 on visual metrics for the site https://cnn.com

For instance, the median SpeedIndex in Chrome was 2662, while in Fenix with Strict tracking protection it was found to be 3447 and 4164 with tracking protection disabled.

Results here (cold page load, live sites, Pixel 3)

This was also seen in a July 2019 visual metrics pass, Pixel 2.

This video compares Chrome and Fenix with Tracking Protection off and strict mode.

Profile of Fenix with Tracking Protection off

Comment 1

[Denis Palmeiro [:denispal]]

This may be due to baseline Jit, see the data in bug 1576202. We'll need tracelogger functional again to investigate further.

Comment 2

[Sean Feng [:sefeng]]

We ran some browsertime tests against different Ion tiers, and it turned out the compiler wasn't the problem

Detail results
https://docs.google.com/spreadsheets/d/169NBGzMX8nFP7oN0RQgy18OwjcL9EHIG5c4oZCDn22E/edit?usp=sharing

Comment 3

[Andrew Creskey [:acreskey]]

The latest data I have comparing Fenix and Chrome on https://www.cnn.com is from Fenix Beta, Dec 18, 2020.
https://docs.google.com/spreadsheets/d/18qCiz3SReDgDPwhbYfuDrbnBK1030FuVWGBHWwdgCFY/edit#gid=96892842&range=303:303

While Fenix was faster in visual metrics than Fennec68, Chrome84 was still faster.

Comment 4

[Andrew Creskey [:acreskey]]

I'll find out if this is still the case.

Comment 5

[Andrew Creskey [:acreskey]]

Still seeing Chrome being faster on vismets for the root of cnn.com
https://docs.google.com/spreadsheets/d/18qCiz3SReDgDPwhbYfuDrbnBK1030FuVWGBHWwdgCFY/edit#gid=2085165784&range=39:41
Will bring to the next Joy of Profiling office hours

Comment 6

[Sean Feng [:sefeng]]

Unassign myself as I am not actively working on it.

Comment 7

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Spoke with acreskey: we noticed that a performance hit seems to occur around the time paint suppression is disabled (iirc, which is a short delay, like ~5s). The next step is to validate that paint suppression is the cause: we can run an experiment where we increase paint suppression (nglayout.initialpaint.delay & nglayout.initialpaint.delay_in_oopif) and compare the results.

Comment 8

[Patricia Lawless]

Performance P1 should also be S2 severity (at least).

Comment 9

[Frank Doty [:fdoty]]

The Performance Priority Calculator has determined this bug's performance priority to be P1. If you'd like to request re-triage, you can reset the Performance flag to "?" or needinfo the triage sheriff.

Platforms: Android
Impact on site: Causes noticeable jank
Page load impact: Some
Websites affected: Major

Comment 10

[Andrew Creskey [:acreskey]]

The raptor-browsertime live site tests are not running for cnn.com on Chrome .
However looking at the cnn-amp results, Chrome is significantly faster on vismets.

https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=8bdcd80751cdce6f835351ea40fa7e2667143e60

I didn't seen any oddities via a quick look at the recorded videos, so this is likely still a problem.

Comment 11

[Markus Stange [:mstange]]

Jamie, could you create a comparison report of loading cnn.com?

Comment 12

[Markus Stange [:mstange]]

Here are some profiles captured in CI on this push:
cold cold cold, warm warm warm

Comment 13

[Markus Stange [:mstange]]

One thing I noticed in these profiles is over 200ms of main thread script compilations, for these files:

114ms ScriptCompileMainThread - <script src="https://edition.cnn.com/.a/2.258.3/js/cnn-header-second-react.min.js ">
38ms ScriptCompileMainThread - <script src="https://edition.cnn.com/optimizelyjs/128727546.js ">
97ms ScriptCompileMainThread - <script src="https://edition.cnn.com/.a/2.258.3/js/cnn-footer-lib-react.min.js ">

Comment 14

[Emilio Cobos Álvarez (:emilio)]

The Performance Impact Calculator has determined this bug's performance impact to be medium. If you'd like to request re-triage, you can reset the Performance Impact flag to "?" or needinfo the triage sheriff.

Platforms: Android
Impact on site: Causes noticeable jank
Page load impact: Some
[x] Able to reproduce locally

Comment 15

[Markus Stange [:mstange]]

The profiles in comment 12 were from recorded page loads.

Here are some profiles captured in CI from live cnn.com, specifically from this "btime-live" job:
cold cold cold, warm warm warm

These also show long ScriptCompileMainThread runnables, just with different script URLs.

Comment 16

[Frank Doty [:fdoty]]

Awaiting work to be reviewed on dependencies

Comment 17

[Marco Castelluccio [:marco]]

Given the performance impact has been retriaged as medium, should we decrease severity?