Closed
Bug 1282120
Opened 9 years ago
Closed 9 years ago
[e10s] Beta 49 e10s-addons experiment (phase 1)
Categories
(Firefox :: Extension Compatibility, defect)
Firefox
Extension Compatibility
Tracking
()
RESOLVED
FIXED
Firefox 50
People
(Reporter: shell, Assigned: Felipe)
References
Details
(Whiteboard: experiment, throttleing, target, triaged)
User Story
First experiment will be to see how/if addons we believe are safe impact performance and stability of Firefox with and without e10s (in Firefox 49 Beta). same criteria that applies to the existing e10s test/control populations (i.e., no a11y, no rtl), minus the no-addons criteria of course, which will follow what is specified here: Beta 49 experiment population (using e10s) has safe addons ("experiment list") defined by ID's and versions (specific few - target # 5 max) and also any addons built using only webextensions control (not using e10s): Same criteria as experiment (specific add-on ID's and version and webextensions) except for users without e10s turned on. experiment list for Beta 49 = https://docs.google.com/spreadsheets/d/1zE1TNV0z6gaYgugBAns3XXDexLDHAa2Dmj0EVQIg8tc/edit#gid=0 Note: both test and control groups for this experiment (which let's call "addons-test" and "addons-control") will come from users who are currently on the "disqualified" cohorts on the existing experiment, because those are the users who have add-ons. We can't use the existing "control" group because the users there don't have any add-ons installed.
Attachments
(1 file)
58 bytes,
text/x-review-board-request
|
mconley
:
review+
Sylvestre
:
approval-mozilla-aurora+
|
Details |
First experiment will be to see how/if addons we believe are safe impact performance of e10s (in a Firefox 49 Beta).
Beta 49 experiment population (using e10s) has safe addons ("allow list) defined by ID's (specific few - target # 5 max) and also any addons built using only webextensions
control (not using e10s): Same criteria as experiment (specific add-ons and webextensions) except for users without e10s turned on.
Comment 1•9 years ago
|
||
I'd like to make sure we're doing rolling comparisons as we roll these out to ensure core assumptions/comparisons don't change release-to-release. As I understand it, in Fx48 we're creating a benchmark where we'll be comparing e10s vs. a control of non-e10s for users with a11y and addons turned off. Are we going to be keeping that control for the duration of the experiments (across releases) so we can continue to analyse against a control as well as the different groups?
e.g. for the first experiment we'd use non-e10s users (no addons, no a11y) as a control, and use non-e10s users with addons and e10s users (no addons, no a11y) to compare against to ensure we're not seeing changes in releases impacting base assumptions.
Will we be maintaining these data points for the duration?
Updated•9 years ago
|
Flags: needinfo?(felipc)
Updated•9 years ago
|
tracking-e10s:
--- → ?
Reporter | ||
Comment 2•9 years ago
|
||
the versions will be specific as well....
Assignee | ||
Comment 3•9 years ago
|
||
If I understood the question right: yeah, we'll be using the same criteria that applies to the existing e10s test/control populations (i.e., no a11y, no rtl), minus the no-addons criteria of course, which will follow the whitelist specified here.
Note: both test and control groups for this experiment (which let's call "addons-test" and "addons-control") will come from users who are currently on the "disqualified" cohorts on the existing experiment, because those are the users who have add-ons. We can't use the existing "control" group because the users there don't have any add-ons installed.
Assignee: nobody → felipc
Status: NEW → ASSIGNED
Flags: needinfo?(felipc)
Reporter | ||
Updated•9 years ago
|
User Story: (updated)
Whiteboard: experiment → experiment, throttleing, target, triaged
Reporter | ||
Comment 5•9 years ago
|
||
finalizing list for initial testing (beyond webextensions) - Kev will let Felipe know when it's got all needed info. https://docs.google.com/spreadsheets/d/1vymALxvAIEAKfEAj-FqFDnQMakZDii05g08qc1T_4gI/edit#gid=0
Reporter | ||
Comment 6•9 years ago
|
||
adding notes from email to Felipe - so it's in one place:
If we wanted 9 addons on the "allow list" in addition to all webextensions - will having 9 versus a smaller number of add-ons impact system add-on performance?
We are going through the final list and 6 are there because they are top addons with many users - 3 are because they will be in our discovery pane and having them on the "allow list" would be useful to avoid the unfriendly restart. Unfriendly UX wise, if people get kicked out of e10s cohort for installing from Discovery Pane and need install.
For the Beta we could pick the ones that get us the most users for testing, but it would be great if there's not a negative to testing discovery pane addons at same tie.
We know 3 for certain if having some for testing helps (working out last approvals for others)...
Addon Name Version ID Link
Greasemonkey 3.8 {e4a8a97b-f2ed-450b-b12d-ee082ba24781} https://addons.mozilla.org/addon/greasemonkey
Download YouTube Videos as MP4 1.8.7 {b9bfaf1c-a63f-47cd-8b9a-29526ced9060} https://addons.mozilla.org/addon/download-youtube
Video download helper 5.6.1 {b9db16a4-6edc-47ec-a1f4-b86292ed211d} https://addons.mozilla.org/addon/video-downloadhelper
Comment 7•9 years ago
|
||
A list of 9 should not materially change the performance of the system add-on. Worry not.
It also shouldn't hurt the initial A/B analysis of "Do users with only addons from this list have a terrible time when e10s is on" since it just defines the two cohorts differently.
Reporter | ||
Updated•9 years ago
|
User Story: (updated)
Flags: needinfo?(sescalante)
Reporter | ||
Comment 8•9 years ago
|
||
just needinfo'ing you to let you know that we're good to move forward with this "allow list" of 9 addons (specific guid and version) in addition to webextensions. only thing that might happen is removing add-ons before beta (working hard not to need that, but possible).
so let me know if there is anything blocking. https://docs.google.com/spreadsheets/d/1vymALxvAIEAKfEAj-FqFDnQMakZDii05g08qc1T_4gI/edit#gid=0
Flags: needinfo?(felipc)
Assignee | ||
Comment 9•9 years ago
|
||
Try builds with the two preliminary set lists (49 beta and 49 release) from the doc above.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ef1fe856d2b
If everything goes well on try we can start QA'ing these builds
Flags: needinfo?(felipc)
Assignee | ||
Updated•9 years ago
|
Summary: First e10s-addons experiment that needs system add-on capability to determine cohort → [e10s] Beta 49 e10s-addons experiment (phase 1)
Assignee | ||
Comment 10•9 years ago
|
||
Review commit: https://reviewboard.mozilla.org/r/65740/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/65740/
Attachment #8773101 -
Flags: review?(mconley)
Assignee | ||
Updated•9 years ago
|
Comment 11•9 years ago
|
||
Comment on attachment 8773101 [details]
Bug 1282120 - Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions).
https://reviewboard.mozilla.org/r/65740/#review62968
Attachment #8773101 -
Flags: review?(mconley) → review+
Comment 12•9 years ago
|
||
Pushed by felipc@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/5ef9e319b525
Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions). r=mconley
Comment 13•9 years ago
|
||
bugherder |
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 50
Assignee | ||
Comment 14•9 years ago
|
||
Comment on attachment 8773101 [details]
Bug 1282120 - Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions).
Approval Request Comment
[Feature/regressing bug #]: this patch actually activates the mechanism from bug 1247497 on Beta, in order to run the e10s-addons experiment on beta 49
[User impact if declined]: no e10s-addons experiment
[Describe test coverage new/current, TreeHerder]: landed on central
[Risks and why]: same as bug 1247497
[String/UUID change made/needed]: none
Attachment #8773101 -
Flags: approval-mozilla-aurora?
Assignee | ||
Comment 15•9 years ago
|
||
And this is the patch that activates the experiment
Flags: needinfo?(sledru)
Updated•9 years ago
|
status-firefox49:
--- → affected
Flags: needinfo?(sledru)
Comment 16•9 years ago
|
||
Comment on attachment 8773101 [details]
Bug 1282120 - Run the e10s-addons experiment on Beta 49 with the set of first 9 add-ons to be tested (plus WebExtensions).
Needed for 49.
Attachment #8773101 -
Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Assignee | ||
Comment 17•9 years ago
|
||
Comment 18•9 years ago
|
||
Pushed by felipc@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4d9e76e287a6
Include Personas Plus in the first e10s-addons experiment for Beta 49. r=me a=me
Assignee | ||
Comment 19•9 years ago
|
||
I included Personas Plus as it was requested by Shell and tested by QA:
https://hg.mozilla.org/integration/mozilla-inbound/rev/4d9e76e287a6
https://hg.mozilla.org/releases/mozilla-aurora/rev/1a408f5eb42e
Comment 20•9 years ago
|
||
bugherder |
Reporter | ||
Updated•9 years ago
|
User Story: (updated)
You need to log in
before you can comment on or make changes to this bug.
Description
•