Closed Bug 1278356 Opened 6 years ago Closed 6 years ago

[e10s] Beta 48 A/B experiment


(Firefox :: General, defect)

48 Branch
Not set



Tracking Status
e10s + ---


(Reporter: Felipe, Assigned: Felipe)




(Whiteboard: [fce-active-legacy])

This bug is to track the e10s experiment running on Beta 48, and to eventually terminate it when decided (thus closing this bug).

The experiment started on beta 1, using the system add-on. It's running at a 50% split on test/control, and the following eligibility filters are present:
 - no add-ons
 - no accessibility (windows, osx, and soon linux when bug 1277882 is uplifted)
 - no RTL languages
 - no OS X 10.6 - 10.8 (bug 1275040)
 - WinXP users are running on beta but should be excluded from any analyses as they will be excluded from release (bug 1275039)
 - GTK 3.20+ users who were excluded from the last experiment are now reincluded (bug 1272751)

This yields approximately 20% of the beta user base in the test group.

The code now allows specifying non-integer cohort thresholds (bug 1268921)

The experiment will auto-start from beta 1 as the setting for beta=50% was already in the tree before uplift.

The previous experiment was tracked on bug 1268627.
From here are the Beta48 proportions:

[(u'disqualified-control', 840161, '30.63%'),
 (u'unknown', 2816, '0.10%'),
 (u'optedOut', 13579, '0.50%'),
 (u'unsupportedChannel', 202, '0.01%'),
 (u'disqualified-test', 840672, '30.65%'),
 (u'optedIn', 2586, '0.09%'),
 (u'control', 527035, '19.21%'),
 (u'disqualified', 48, '0.00%'),
 (u'test', 515982, '18.81%')]

Notably different, in that the control/test groups lost a few %ge points each to their disqualified branches. Which of the criteria might be responsible, I wonder?
The experiment ran through the entire beta cycle, and the links to the weekly analysis can be found at [1]. The final weekly analysis is at [2].


All of e10s release criteria were met during this experiment, and e10s-no-addons is going out to release on 48.
Closed: 6 years ago
Resolution: --- → FIXED
Whiteboard: [fce-active] → [fce-active-legacy]
You need to log in before you can comment on or make changes to this bug.