1620271 - Use a predefiend set of performance prefs in mach browsertime

Status: NEW

(Testing :: Performance, task, P3)

Description

[Greg Mierzwinski [:sparky]]

This bug is for adding the usage of a predefined set of performance prefs in mach browsertime. At the moment, there are no prefs being set and no profile is provided to it.

These should ideally be upstreamed to sitespeedio as well and they should be usable there with an optional flag (to use those prefs instead of the default ones).

:jesup, :denispal can one of you either provide the prefs needed here or make a patch for this issue?

Comment 1

[Andrew Creskey [:acreskey]]

Greg, by mach browsertime I assume you mean the userflow where someone clones a repo and runs browsertime tests locally?
e.g.
./mach browsertime https://www.mozilla.org?

In that case I'm pretty sure that the browsertime prefs would be used, i.e those in `firefoxPreferences.js'
https://github.com/sitespeedio/browsertime/blob/d0048bf3f79d04e8617f2ee5a8549049416fea8e/lib/firefox/webdriver/firefoxPreferences.js

This is a very significant set of prefs, including such things as disabling Tracking Protection.
(See https://github.com/sitespeedio/browsertime/issues/1211)

There are also a set of performance-significant prefs that are only set in browsertime when running raptor-browsertime (e.g. firefox.disableBrowsertimeExtension)
https://searchfox.org/mozilla-central/rev/4d9cd186767978a99dafe77eb536a9525980e118/testing/raptor/raptor/browsertime/base.py#194

Ideally we can come up with a solution where ./mach browsertime and also browsertime perf tests use the same set of prefs.

Comment 2

[Greg Mierzwinski [:sparky]]

:acreskey, I agree - I'm wondering if we can work out a few different sets of prefs here: (1) for performance, (2) for stability, (3) balanced/combination of the two previous ones, (4) for realism (very minimal prefs?).

As you mentioned, we have this list of prefs used now, and we need to cut out what we shouldn't be setting: https://github.com/sitespeedio/browsertime/blob/d0048bf3f79d04e8617f2ee5a8549049416fea8e/lib/firefox/webdriver/firefoxPreferences.js

I need help from yourself, :denispal, and :jesup to build these lists. If either of you could provide a few lists as a starting point that would be great! We could consider the tracking protection disabling as an extra set of prefs that gets applied to the other sets.

Comment 3

[Denis Palmeiro [:denispal]]

These are the prefs we settled on previously to imitate reality as close as possible : https://github.com/mozilla/browsertime/blob/master/lib/firefox/webdriver/firefoxPreferences.js

Comment 4

[Andrew Creskey [:acreskey]]

(In reply to Greg Mierzwinski [:sparky] from comment #2)

... We could consider the tracking protection disabling as an extra set of prefs that gets applied to the other sets.

Yes, I like having TrackingProtection/SafeBrowsing as separate options.
Peter has already extended Denis' initial configuration of these prefs here https://github.com/sitespeedio/browsertime/pull/1272
--firefox.disableSafeBrowsing false --firefox.disableTrackingProtection false

(In reply to Greg Mierzwinski [:sparky] from comment #2)

:acreskey, I agree - I'm wondering if we can work out a few different sets of prefs here: (1) for performance, (2) for stability, (3) balanced/combination of the two previous ones, (4) for realism (very minimal prefs?).

I don't know that I could directly tell you which prefs would be assigned to each of these categories.
For realism, I would also favor the set that Denis provided in Comment 3 as well as the option to enable/disable TP/SafeBrowsing.