1278356 - [e10s] Beta 48 A/B experiment

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

This bug is to track the e10s experiment running on Beta 48, and to eventually terminate it when decided (thus closing this bug).

The experiment started on beta 1, using the system add-on. It's running at a 50% split on test/control, and the following eligibility filters are present:
 - no add-ons
 - no accessibility (windows, osx, and soon linux when bug 1277882 is uplifted)
 - no RTL languages
 - no OS X 10.6 - 10.8 (bug 1275040)
 - WinXP users are running on beta but should be excluded from any analyses as they will be excluded from release (bug 1275039)
 - GTK 3.20+ users who were excluded from the last experiment are now reincluded (bug 1272751)

This yields approximately 20% of the beta user base in the test group.

The code now allows specifying non-integer cohort thresholds (bug 1268921)

The experiment will auto-start from beta 1 as the setting for beta=50% was already in the tree before uplift.

The previous experiment was tracked on bug 1268627.

https://wiki.mozilla.org/Electrolysis/Experiments

Comment 1

[Chris H-C :chutten]

From https://github.com/vitillo/e10s_analyses/tree/master/beta/48/week1-2 here are the Beta48 proportions:

[(u'disqualified-control', 840161, '30.63%'),
 (u'unknown', 2816, '0.10%'),
 (u'optedOut', 13579, '0.50%'),
 (u'unsupportedChannel', 202, '0.01%'),
 (u'disqualified-test', 840672, '30.65%'),
 (u'optedIn', 2586, '0.09%'),
 (u'control', 527035, '19.21%'),
 (u'disqualified', 48, '0.00%'),
 (u'test', 515982, '18.81%')]

Notably different, in that the control/test groups lost a few %ge points each to their disqualified branches. Which of the criteria might be responsible, I wonder?

Comment 2

[:Felipe Gomes (needinfo for replies!)]

Ok, I looked into telemetry [1] and there's 3.9% users hitting criteria #9 [2], so the answer is bug 1237769, which disabled e10s for XP users running graphics acceleration.

[1] https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-06-20&keys=__none__!__none__!__none__&max_channel_version=beta%252F48&measure=E10S_STATUS&min_channel_version=null&processType=false&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-06-06&table=0&trim=1&use_submission_date=0

[2] http://searchfox.org/mozilla-central/rev/ef24c234ed53b3ba50a1734f6b946942e4434b5b/toolkit/xre/nsAppRunner.cpp#4718

Comment 3

[:Felipe Gomes (needinfo for replies!)]

The experiment ran through the entire beta cycle, and the links to the weekly analysis can be found at [1]. The final weekly analysis is at [2].

[1] https://wiki.mozilla.org/Electrolysis/Experiments#Beta_48
[2] https://github.com/vitillo/e10s_analyses/blob/master/beta/48/week7/e10s_experiment.ipynb

All of e10s release criteria were met during this experiment, and e10s-no-addons is going out to release on 48.