1282120 - [e10s] Beta 49 e10s-addons experiment (phase 1)

Status: RESOLVED FIXED

(Firefox :: Extension Compatibility, defect)

Description

[:shell escalante]

First experiment will be to see how/if addons we believe are safe impact performance of e10s (in a Firefox 49 Beta).  

Beta 49 experiment population (using e10s) has safe addons ("allow list) defined by ID's (specific few - target # 5 max) and also any addons built using only webextensions

control (not using e10s): Same criteria as experiment (specific add-ons and webextensions) except for users without e10s turned on.

Comment 1

[Kev Needham [:kev]]

I'd like to make sure we're doing rolling comparisons as we roll these out to ensure core assumptions/comparisons don't change release-to-release. As I understand it, in Fx48 we're creating a benchmark where we'll be comparing e10s vs. a control of non-e10s for users with a11y and addons turned off. Are we going to be keeping that control for the duration of the experiments (across releases) so we can continue to analyse against a control as well as the different groups?

e.g. for the first experiment we'd use non-e10s users (no addons, no a11y) as a control, and use non-e10s users with addons and e10s users (no addons, no a11y) to compare against to ensure we're not seeing changes in releases impacting base assumptions.

Will we be maintaining these data points for the duration?

Comment 2

[:shell escalante]

the versions will be specific as well....

Comment 3

[:Felipe Gomes (needinfo for replies!)]

If I understood the question right: yeah, we'll be using the same criteria that applies to the existing e10s test/control populations (i.e., no a11y, no rtl), minus the no-addons criteria of course, which will follow the whitelist specified here.

Note: both test and control groups for this experiment (which let's call "addons-test" and "addons-control") will come from users who are currently on the "disqualified" cohorts on the existing experiment, because those are the users who have add-ons. We can't use the existing "control" group because the users there don't have any add-ons installed.

Comment 4

[:shell escalante]

Shell get add-on list draft out

Comment 5

[:shell escalante]

finalizing list for initial testing (beyond webextensions) - Kev will let Felipe know when it's got all needed info.  https://docs.google.com/spreadsheets/d/1vymALxvAIEAKfEAj-FqFDnQMakZDii05g08qc1T_4gI/edit#gid=0

Comment 6

[:shell escalante]

adding notes from email to Felipe - so it's in one place:

If we wanted 9 addons on the "allow list" in addition to all webextensions - will having 9 versus a smaller number of add-ons impact system add-on performance?

We are going through the final list and 6 are there because they are top addons with many users - 3 are because they will be in our discovery pane and having them on the "allow list" would be useful to avoid the unfriendly restart.  Unfriendly UX wise, if people get kicked out of e10s cohort for installing from Discovery Pane and need install.

For the Beta we could pick the ones that get us the most users for testing, but it would be great if there's not a negative to testing discovery pane addons at same tie.

We know 3 for certain if having some for testing helps (working out last approvals for others)...
Addon Name	Version	   ID	   Link
Greasemonkey 3.8	 {e4a8a97b-f2ed-450b-b12d-ee082ba24781}	https://addons.mozilla.org/addon/greasemonkey
Download YouTube Videos as MP4 1.8.7 {b9bfaf1c-a63f-47cd-8b9a-29526ced9060} https://addons.mozilla.org/addon/download-youtube
Video download helper 5.6.1 {b9db16a4-6edc-47ec-a1f4-b86292ed211d} https://addons.mozilla.org/addon/video-downloadhelper

Comment 7

[Chris H-C :chutten]

A list of 9 should not materially change the performance of the system add-on. Worry not.

It also shouldn't hurt the initial A/B analysis of "Do users with only addons from this list have a terrible time when e10s is on" since it just defines the two cohorts differently.

Comment 8

[:shell escalante]

just needinfo'ing you to let you know that we're good to move forward with this "allow list" of 9 addons (specific guid and version) in addition to webextensions.  only thing that might happen is removing add-ons before beta (working hard not to need that, but possible). 

so let me know if there is anything blocking.  https://docs.google.com/spreadsheets/d/1vymALxvAIEAKfEAj-FqFDnQMakZDii05g08qc1T_4gI/edit#gid=0

Comment 9

[:Felipe Gomes (needinfo for replies!)]

Try builds with the two preliminary set lists (49 beta and 49 release) from the doc above.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ef1fe856d2b

If everything goes well on try we can start QA'ing these builds

Comment 10

[:Felipe Gomes (needinfo for replies!)]

Attachment: Bug 1282120 - Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions).

Review commit: https://reviewboard.mozilla.org/r/65740/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65740/

Comment 11

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8773101 [details]
Bug 1282120 - Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions).

https://reviewboard.mozilla.org/r/65740/#review62968

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5ef9e319b525
Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions). r=mconley

Comment 13

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/5ef9e319b525

Comment 14

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8773101 [details]
Bug 1282120 - Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions).

Approval Request Comment
[Feature/regressing bug #]: this patch actually activates the mechanism from bug 1247497 on Beta, in order to run the e10s-addons experiment on beta 49
[User impact if declined]: no e10s-addons experiment
[Describe test coverage new/current, TreeHerder]: landed on central
[Risks and why]: same as bug 1247497
[String/UUID change made/needed]: none

Comment 15

[:Felipe Gomes (needinfo for replies!)]

And this is the patch that activates the experiment

Comment 16

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8773101 [details]
Bug 1282120 - Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions).

Needed for 49.

Comment 17

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/29c64bded0f1

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4d9e76e287a6
Include Personas Plus in the first e10s-addons experiment for Beta 49. r=me a=me

Comment 19

[:Felipe Gomes (needinfo for replies!)]

I included Personas Plus as it was requested by Shell and tested by QA:

https://hg.mozilla.org/integration/mozilla-inbound/rev/4d9e76e287a6
https://hg.mozilla.org/releases/mozilla-aurora/rev/1a408f5eb42e

Comment 20

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4d9e76e287a6