1297755 - (system-addon-e10s) Beta 50 - E10s add-ons A/B test with all multiprocessCompatible add-ons

Status: RESOLVED FIXED

(Firefox :: Extension Compatibility, defect, P1)

Description

[:shell escalante]

As Beta 49 is going well - we'd want to increase the audience.  

not sure what the best options for design:
two options - 
1.  can we do a huge list with 500 or so add-ons marked MPC:true (and we manually exclude ones we know have bugs)
2.  can we include all mpc:true that are not marked with having a bug against them? (there is a json file that goes off ID and bug number - but doesn't show status of bug in json.

Comment 1

[:shell escalante]

looking at the user story - is option 1 a viable option?  not sure if list size is a limiting factor for performance.

Comment 2

[Jim Mathies [:jimm]]

One important detail here for Fx50, we'll have a low integrity sandbox (system file write access restrictions) in place during this experiment. What we would like to do is see how the data looks over the first few weeks with htis in place. If we see issues that may be sandboxing related, we will want to flip a pref mid-experiment to turn sandboxing off. Then we can compare the data. The add-ons team will make this call based on what we're seeing.

Felipe, is this doable with our planned system addon for 50?

Comment 3

[:Felipe Gomes (needinfo for replies!)]

Yeah, we should just set the sandboxing-related prefs normally on the tree, unrelated to the system addon. Whenever we flip the pref in the middle of the cycle, we can at the same time change the cohort name so that it's easier to differentiate when looking at the data

Comment 4

[:Felipe Gomes (needinfo for replies!)]

What should be the initial state of sandboxing-related prefs when Beta 50 starts? Is it already properly set in mozilla-aurora?

Comment 5

[:Felipe Gomes (needinfo for replies!)]

Attachment: Bug 1297755 - Allow e10s add-on policy based on the multiprocessCompatible flag.

Review commit: https://reviewboard.mozilla.org/r/78204/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/78204/

Comment 6

[:Felipe Gomes (needinfo for replies!)]

Attachment: Bug 1297755 - Configure Beta 50 to run an e10s A/B experiment with all multiprocessCompatible addons.

Review commit: https://reviewboard.mozilla.org/r/78206/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/78206/

Comment 7

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8790914 [details]
Bug 1297755 - Allow e10s add-on policy based on the multiprocessCompatible flag.

https://reviewboard.mozilla.org/r/78204/#review77276

Comment 8

[Mike Conley (:mconley) (:⚙️)]

Comment on attachment 8790915 [details]
Bug 1297755 - Configure Beta 50 to run an e10s A/B experiment with all multiprocessCompatible addons.

https://reviewboard.mozilla.org/r/78206/#review77282

Thanks felipe! One question - see below.

::: browser/extensions/e10srollout/bootstrap.js:73
(Diff revision 1)
> +    Preferences.set(PREF_E10S_ADDON_BLOCKLIST,
> +                    "{dc572301-7619-498c-a57d-39143191b318}");

Out of curiosity, what's the plan for clearing this once bug 1185672 is resolved?

Comment 9

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8790915 [details]
Bug 1297755 - Configure Beta 50 to run an e10s A/B experiment with all multiprocessCompatible addons.

https://reviewboard.mozilla.org/r/78206/#review77332

::: browser/extensions/e10srollout/bootstrap.js:73
(Diff revision 1)
> +    Preferences.set(PREF_E10S_ADDON_BLOCKLIST,
> +                    "{dc572301-7619-498c-a57d-39143191b318}");

I plan to replace this with a Preferences.reset(PREF_E10S_ADDON_BLOCKLIST). Or in case we need to have another add-on blocklisted, just changing the string to override it.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a6637618e917
Allow e10s add-on policy based on the multiprocessCompatible flag. r=mconley
https://hg.mozilla.org/integration/autoland/rev/e5195a75e9e7
Configure Beta 50 to run an e10s A/B experiment with all multiprocessCompatible addons. r=mconley

Comment 11

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/a6637618e917
https://hg.mozilla.org/mozilla-central/rev/e5195a75e9e7

Comment 12

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8790914 [details]
Bug 1297755 - Allow e10s add-on policy based on the multiprocessCompatible flag.

Approval Request Comment
[Feature/regressing bug #]: E10s add-ons A/B test on Beta 50: all multiprocessCompatible add-ons
[User impact if declined]: won't run the test on next Beta as we want
[Describe test coverage new/current, TreeHerder]: landed on central
[Risks and why]: risk of exposing users to e10s and new add-ons
[String/UUID change made/needed]: none

Comment 13

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8790915 [details]
Bug 1297755 - Configure Beta 50 to run an e10s A/B experiment with all multiprocessCompatible addons.

Approval Request Comment
[Feature/regressing bug #]: E10s add-ons A/B test on Beta 50: all multiprocessCompatible add-ons
[User impact if declined]: won't run the test on next Beta as we want
[Describe test coverage new/current, TreeHerder]: landed on central
[Risks and why]: risk of exposing users to e10s and new add-ons
[String/UUID change made/needed]: none

Comment 14

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Felipe, just to better understand the impact, how many end-users do we expect to run this experiment on? Is it 50% of all beta users? less or more? Also what kind of monitoring will we do (like DAU) to look for a negative impact and turn off this experiment? Lastly, how long is this experiment planned to run on Beta50?

Comment 15

[:Felipe Gomes (needinfo for replies!)]

Hi Ritu, I think we don't know the numbers exactly, and part of the purpose is discovering how many users fit into the multiprocessCompatible add-ons criteria. Currently Beta is running with 21% of users on e10s, and a rough estimate is that this will (at most) double that, to 42%.

David Zeber, Shell and the add-ons team will continue to monitor the test and control cohorts in telemetry and crash stats like they did on 49 to check how they compare.

The length of the experiment is tbd, it depends on what we find at the beginning. If everything is looking smooth I know that Shell wants to run a different experiment with all add-ons, but that's separate and will be discussed in a near future.

Comment 16

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Thanks Felipe, the additional details are much appreciated. Let's uplift to Aurora50.

I will add this to my list of Beta50 experiments (might be the only one in that list for now).

Comment 17

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8790914 [details]
Bug 1297755 - Allow e10s add-on policy based on the multiprocessCompatible flag.

Aurora50+

Comment 18

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8790915 [details]
Bug 1297755 - Configure Beta 50 to run an e10s A/B experiment with all multiprocessCompatible addons.

Aurora50+

Comment 19

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/a9c45aee1a0c
https://hg.mozilla.org/releases/mozilla-aurora/rev/8c37b5ac91e8

Comment 20

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Hi Felipe, Shell, Erin, Beta50 crash rate (based on telemetry data) is much worse than Beta49. Could it be due to this experiment? In order to reduce the noise, can we stop this experiment after 50.0b6 so I have a better understanding of the real problems causing Beta50 crash rates to increase? I am at a loss on other ideas and would appreciate your help on this one.

Comment 21

[:shell escalante]

Redirecting to Dave who was getting the Beta 50 links on the experiment - can we get that ASAP to see what impact the add-ons experiment is having?  at least we'll have more data - to see if it's related

Comment 22

[:shell escalante]

David is updating the redash info (hopefully today) for 50 which as the crash stats. it will have same info as 49 redash on crashes https://sql.telemetry.mozilla.org/dashboard/stability-comparison-for-e10s-add-ons-experiment (old 49 Beta info - needs updating for 50 beta)

adding in perf info from python notebook (thank you dave).  https://github.com/dzeber/e10s_analyses/blob/beta-50/beta/50/week1/e10s_experiment.ipynb - "Looks pretty much the same as 49 - no alarm bells. Overall, e10s seems to have improved things."

Comment 23

[:Felipe Gomes (needinfo for replies!)]

Yeah, beta is still doing an A/B split where we can look at half of the users with e10s on and off. So let's do a comparison between the two cohorts to see if the increase in crashes is specific to e10s-addon users

Comment 24

[Dave Zeber [:dzeber]]

Here is the link to the new stability dashboard for this experiment:

 <https://sql.telemetry.mozilla.org/dashboard/stability-metrics-for-e10s-add-ons-experiment-beta-49-50-51-> (note the trailing dash, which may not have gotten included in the hyperlink)

It combines crash rates across Beta 49-51, broken out by addons/no addons and e10s/no e10s. The graphs near the top show crash rates by date of activity (which may include users across multiple builds), and the ones in the middle show crash rates by build date.