Open Bug 1583222 Opened 5 years ago Updated 5 months ago

Visual metrics: Fenix is significantly slower than Chrome for the site https://www.reddit.com

Categories

(Core :: Performance, defect, P2)

ARM64
Android
defect

Tracking

()

Performance Impact medium

People

(Reporter: acreskey, Unassigned, NeedInfo)

Details

(Keywords: perf:pageload, perf:responsiveness, top50)

Attachments

(2 files)

Fenix is significantly slower than Chrome 76 on visual metrics for the site https://www.reddit.com
For instance, the median SpeedIndex in Chrome was 1723, while in Fenix with Strict tracking protection it was found to be 2226 and 2379 with tracking protection disabled.

Results here (cold page load, live sites, Pixel 3)

This was also seen in a July 2019 visual metrics pass, Pixel 2.

This video compares Chrome and Fenix with Tracking Protection off and strict mode.

Profile of Fenix with Tracking Protection off

This is probably a dup of bug 1574024

Assignee: nobody → acreskey
Whiteboard: [qf] → [qf:p1:pageload]
Priority: -- → P1
Assignee: acreskey → sefeng

We ran some browsertime tests against different Ion tiers, and it turned out the compiler wasn't the problem

Detail results
https://docs.google.com/spreadsheets/d/1rd7r70yXjuBF54bjMnIT6bQNM6vd2ejPmEjK_s6zVjc/edit?usp=sharing

As in Bug 1583220, the latest data I have comparing Fenix and Chrome on https://www.reddit.com is from Fenix Beta, Dec 18, 2020.
https://docs.google.com/spreadsheets/d/18qCiz3SReDgDPwhbYfuDrbnBK1030FuVWGBHWwdgCFY/edit#gid=96892842&range=360:360

While Fenix was faster in visual metrics than Fennec68, Chrome84 was still faster on https://www.reddit.com

I'll find out if this is still the case.

Flags: needinfo?(acreskey)

It looks like this is still the case - Chrome is reproducibly faster than Fenix on vismets.
I used this url for content stability:
https://www.reddit.com/r/firefox/

https://docs.google.com/spreadsheets/d/18qCiz3SReDgDPwhbYfuDrbnBK1030FuVWGBHWwdgCFY/edit#gid=623041654&range=34:34

Actually, I haven't been able to reproduce a significant difference between Chrome and Fenix.
https://docs.google.com/spreadsheets/d/18qCiz3SReDgDPwhbYfuDrbnBK1030FuVWGBHWwdgCFY/edit#gid=2085165784&range=42:42
I've captured profiles that we'll examine just to see if anything stands out.

Unassign myself as I am not actively working on it.

Assignee: sefeng → nobody

I took a manual side-by-side videos on different days and got the following results:

Date Chrome Fenix Diff
10/14 5700ms 8600ms 2900ms
11/3 #1 7966ms 7666ms -300ms
11/3 #2 8133ms 6333ms -1800ms

Curiously, Chrome got slower today than the previous test date – I wonder if the page contents or my network conditions changed.

Attachment #9249107 - Attachment description: 10-14-reddit-side-by-side.mp4 → 10/14: reddit-side-by-side.mp4

One major difference between the videos is that Firefox's first paint (to the reddit loading icon) is much faster on 11/3 than it was on 10/14.

Assignee: nobody → acreskey
Flags: needinfo?(acreskey)

Spoke with acreskey: now that we can't reproduce such a large difference, the next step is to check the cause. We should try setting dom.script_loader.full_parse_max_size to 0 and comparing the results to see if the full parse was helpful.

Performance Impact: --- → P1
Keywords: perf:pageload
Whiteboard: [qf:p1:pageload]

Performance P1 should also be S2 severity (at least).

Severity: normal → S2

The Performance Priority Calculator has determined this bug's performance priority to be P1. If you'd like to request re-triage, you can reset the Performance flag to "?" or needinfo the triage sheriff.

Platforms: Android
Impact on site: Causes noticeable jank
Page load impact: Some
Websites affected: Major

Frank, according to comment 11 this was no longer reproducible. Should we close it or re-test?

Flags: needinfo?(fdoty)

:mleclair, is this something that we'd be able to measure going off of mcomella's last update in Comment 11? If so, could be worth tracking for the future. If you feel that this is too old or not reproducible, we can close

Flags: needinfo?(fdoty) → needinfo?(mleclair)

I'm trying to collect some numbers on these via our performance tests since we have support for live sites Fenix and Chrome.
(along with Bug 1583220)

Currently blocked on Bug 1780817, but it looks like it's progressing well.

Our live-site Android tests have been fixed.

Unfortunately this still looks to be an issue.
https://treeherder.mozilla.org/jobs?repo=try&selectedTaskRun=ItAk-jtGSJyY6vQyd02aFQ.0&tier=1%2C2%2C3&revision=8bdcd80751cdce6f835351ea40fa7e2667143e60

Chrome shows reproducibly better visual metrics than Fenix on reddit.com.
From a look at the video recordings, the results are valid.

I suggest we take some of the profiles to JoyOfProfiling sessions to see if we can spot any issues.

Assignee: acreskey → nobody

Jamie, could you create a comparison report of loading reddit.com?

Here are some profiles captured in CI on this push:
cold cold cold, warm warm warm

Performance Impact: high → ?

Not enough data to show that this reproduces locally. Tried this on my Pixel 8 and while cold load was slower (though not significantly), performed just as well on warm load. Calculator shows a High (75) score. Gut feeling is this is a Medium impact.

A profile with more data would be required to make this a more significant impact

The Performance Impact Calculator has determined this bug's performance impact to be high. If you'd like to request re-triage, you can reset the Performance Impact flag to "?" or needinfo the triage sheriff.

Platforms: Android
Page load impact: Some
Websites affected: Major
[x] Able to reproduce locally

Asking :mleclair to grab a profile and a new video comparison

Performance Impact: ? → medium
Priority: P1 → P2

Given the lowered performance impact, we should reduce severity.

Here are some more profiles from CI, from reddit.com live site (specifically from this push):

cold cold cold, warm warm warm

Nothing particularly stands out.

(In reply to Marco Castelluccio [:marco] from comment #21)

Given the lowered performance impact, we should reduce severity.

Frank, do you agree?

Flags: needinfo?(fdoty)
Severity: S2 → S3
Flags: needinfo?(fdoty)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: