1390206 - AWFY - ensure that test profiles exclude experiments

Status: RESOLVED FIXED

(Testing Graveyard :: AWFY, enhancement)

Description

[Bob Clary [:bc] (inactive)]

bkelly asks about whether awfy's profiles exclude experiments such as enabling stylo, etc. Suggests that the experiments are tied to telemetry settings.

wlach suggests that mozprofile would automatically set the appropriate preferences and should be used if it is not already.

I'll look into this asap.

Comment 1

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/arewefastyet/blob/master/slave/configs.py

https://github.com/mozilla/arewefastyet/blob/master/slave/executors.py

show that the prefs are constructed as a string in configs.py that is written to profile/prefs.js in executors.py.

So, we do not use mozprofile and probably don't disable the same set of preferences we do elsewhere in automation.

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: https://github.com/mozilla/arewefastyet/pull/159

The first commit changes from using a string of user_pref values in profile_ to keeping the preferences in a dictionary that is passed to FirefoxProfile to create the profile.

The second commit adds the preference prefs["layout.css.servo.enabled"] = False to prevent servo from running.

We will have to install mozprofile before merging this. I intend to just do a global install via pip.

Comment 3

[Bob Clary [:bc] (inactive)]

Attachment: user.js prefs set by default by FirefoxProfile

This attachment lists the preferences set by default by mozprofile. Note they are stored in the user.js file. The prefs.js file in the profile is initialized to empty by FirefoxProfile.

Additional preferences set via the preferences argument to the constructor will appear below the initial section.

Comment 4

[Ben Kelly [:bkelly, not reviewing]]

Comment on attachment 8897153 [details] [diff] [review]
https://github.com/mozilla/arewefastyet/pull/159

Honestly, I don't know enough about our experiments system or test infrastructure to provide feedback here.  I'm just happy to see it getting looked at.  Thanks!

Comment 5

[Bob Clary [:bc] (inactive)]

One question is: Do we disable stylo or enable it by default. Currently, the patch turns it off.
Another is: Do we want to support both configurations and run both types of job?

Comment 6

[Ben Kelly [:bkelly, not reviewing]]

Until we enable stylo by default I would prefer to have it disabled on AWFY.  Or two separate jobs.  Just my opinion, though.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8897153 [details] [diff] [review]
https://github.com/mozilla/arewefastyet/pull/159

Review of attachment 8897153 [details] [diff] [review]:
-----------------------------------------------------------------

a simple change- hard to look at code that isn't mozbase related anymore- make sure that you install all other mozprofile related packages as well.

Comment 8

[Stuart Philp :sphilp]

I alluded to this in my github comment, but so it's visible here too, this is generally fine for now. I think long term we're going to at least want a simple file to map to for explicitly setting prefs like this, and perhaps one day using a service (shield may even work?) to control our automation configuration state. With a service, we could timebox experimental pref toggling (say for 24 hours) for a desired state to check results, possibly reducing the need for ballooning the number of configurations we run under (as in we'd only ever run 1 configuration, but the state of that configuration could be more dynamic).

Comment 9

[Benjamin Bouvier [:bbouvier] (inactive)]

Comment on attachment 8897153 [details] [diff] [review]
https://github.com/mozilla/arewefastyet/pull/159

Had one comment that I've put on Github, otherwise looks good to me, thanks!

Comment 10

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/arewefastyet/commit/d07fae13804009918a0eeb4f757d7e455afa299d
https://github.com/mozilla/arewefastyet/commit/71cb9664696776ab45daf3ff6610ac87685f56a2

I followed Ben's advice. The idea of using Shield is intriguing but something to look into later rather than now. Thanks everyone!