[e10s] Beta 48 A/B experiment

RESOLVED FIXED

Status

()

Firefox
General
RESOLVED FIXED
2 years ago
3 months ago

People

(Reporter: Felipe, Assigned: Felipe)

Tracking

48 Branch
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(e10s+)

Details

(Whiteboard: [fce-active-legacy], URL)

(Assignee)

Description

2 years ago
This bug is to track the e10s experiment running on Beta 48, and to eventually terminate it when decided (thus closing this bug).

The experiment started on beta 1, using the system add-on. It's running at a 50% split on test/control, and the following eligibility filters are present:
 - no add-ons
 - no accessibility (windows, osx, and soon linux when bug 1277882 is uplifted)
 - no RTL languages
 - no OS X 10.6 - 10.8 (bug 1275040)
 - WinXP users are running on beta but should be excluded from any analyses as they will be excluded from release (bug 1275039)
 - GTK 3.20+ users who were excluded from the last experiment are now reincluded (bug 1272751)

This yields approximately 20% of the beta user base in the test group.

The code now allows specifying non-integer cohort thresholds (bug 1268921)

The experiment will auto-start from beta 1 as the setting for beta=50% was already in the tree before uplift.

The previous experiment was tracked on bug 1268627.

https://wiki.mozilla.org/Electrolysis/Experiments

Comment 1

2 years ago
From https://github.com/vitillo/e10s_analyses/tree/master/beta/48/week1-2 here are the Beta48 proportions:

[(u'disqualified-control', 840161, '30.63%'),
 (u'unknown', 2816, '0.10%'),
 (u'optedOut', 13579, '0.50%'),
 (u'unsupportedChannel', 202, '0.01%'),
 (u'disqualified-test', 840672, '30.65%'),
 (u'optedIn', 2586, '0.09%'),
 (u'control', 527035, '19.21%'),
 (u'disqualified', 48, '0.00%'),
 (u'test', 515982, '18.81%')]

Notably different, in that the control/test groups lost a few %ge points each to their disqualified branches. Which of the criteria might be responsible, I wonder?
(Assignee)

Comment 3

2 years ago
The experiment ran through the entire beta cycle, and the links to the weekly analysis can be found at [1]. The final weekly analysis is at [2].

[1] https://wiki.mozilla.org/Electrolysis/Experiments#Beta_48
[2] https://github.com/vitillo/e10s_analyses/blob/master/beta/48/week7/e10s_experiment.ipynb

All of e10s release criteria were met during this experiment, and e10s-no-addons is going out to release on 48.
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED

Updated

3 months ago
Whiteboard: [fce-active] → [fce-active-legacy]
You need to log in before you can comment on or make changes to this bug.