1592633 - Handling differences in prefs between browsertime and raptor-browsertime

Status: NEW

(Testing :: Raptor, task, P3)

Description

[Greg Mierzwinski [:sparky]]

This bug is to discuss what we should do in terms of handling pref settings across browsertime, and raptor-browsertime. There are three pref setting options: (i) browsertime, (ii) perf-profile (raptor-browsertime settings), or (iii) browsertime+perf-profile.

One option, as :acreskey suggested, is the following:

One way to allow the 3 sets of prefs (browsertime, perf profile, browsertime+perf profile) to work consistenly:
We could add a flag to browsertime to ignore the default firefoxPreferences.js
Denis has already done something similar to provide an easy way to disable safebrowsing:
https://github.com/mozilla/browsertime/blob/master/lib/firefox/webdriver/index.js#L137
Similarly, we could skip the default pref injection here with an option flag:
https://github.com/mozilla/browsertime/blob/master/lib/firefox/webdriver/index.js#L76

Comment 1

[Andrew Creskey [:acreskey]]

As Florin pointed out, the browsertime runs also have additional prefs injected.

Through browsertime
https://github.com/mozilla/browsertime/blob/master/lib/firefox/webdriver/index.js#L137

And also these:

Geckodriver:
https://searchfox.org/mozilla-central/source/testing/geckodriver/src/prefs.rs#16
These look to have at least a small impact on raptor-tp6-wikipedia-firefox-cold , windows10-64-shippable
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=f8a86eaf23dc0b5fec01af18a8c2a487c551a119&newProject=try&newRevision=ca2a2864e2d6fc7ba32ea802cb02304ed946cacc&framework=10

Marionette:
https://searchfox.org/mozilla-central/source/testing/marionette/components/marionette.js#68
I'll add the above to a raptor push to see if they are impactful.

Comment 2

[Andrew Creskey [:acreskey]]

This is the impact of adding the marionette prefs to raptor:

https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=f8a86eaf23dc0b5fec01af18a8c2a487c551a119&newProject=try&newRevision=f6bc22a9f7b73684b79134c8d8fd2ec92f680899&framework=10

The only change marked as significant by perfherder is a 2% improvement on raptor-tp6-bing-firefox-cold on OSX.
In my opinion, I think we can live with that.

Comment 3

[Greg Mierzwinski [:sparky]]

First part of this work is will be done in bug 1620271.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

Mass-removing myself from cc; search for 12b9dfe4-ece3-40dc-8d23-60e179f64ac1 or any reasonable part thereof, to mass-delete these notifications (and sorry!)