Closed Bug 1225496 Opened 9 years ago Closed 8 years ago

Beta45 e10s experiment: split the beta population in two for e10s testing

Categories

(Core :: DOM: Content Processes, defect)

Product:
Core
Component:
DOM: Content Processes
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

()

Status:
RESOLVED DUPLICATE of bug 1241336
Tracking Flags:
Tracking Status
e10s m8+ ---
firefox45 + ---

People

(Reporter: benjamin, Assigned: vladan)

References

Details

User Story

Upon install/upgrade of beta 44 (elan: confirm), each profile should randomly select to be in one of two groups:

CONTROL GROUP - no e10s

The control group will not have e10s enabled and will continue to run the release non-e10s configuration.

TEST GROUP - e10s

The test group will be placed into the e10s configuration which we are most likely to ship. e10s will continue to be disabled for users with accessibility but on for everyone else.

Technical requirements:
* there must not be any preferences option to disable or enable e10s or change groups
* there should be no "open in non-e10s window" item available in the Firefox menu
* Users should be able to determine which group they are in by visiting about:support
* Unified telemetry and crash reports should record which group users were put in
* The tab titles should not include "e10s"

OPEN QUESTIONS:
* should users in the e10s group be able to disable e10s via some temporary UI for testing purposes?
In order to ship e10s, we need to have widespread testing of the e10s codepaths. We also need to maintain widespread testing of the non-e10s codepaths so that we can keep shipping releases without e10s. 

This represents a decision in the last e10s meeting I attended. Erin, I'd like you to double-check my description and clarify questions in the user story and make sure we have the right product and release management approvals to ship this.
Flags: needinfo?(elancaster)
"there must not be any preferences option to disable or enable e10s or change groups" sounds like we forcibly take user control away. How does that fit with our overall messaging that we want to improve user control?

Also, can we first get results from a smaller set of people and fix the issues we know make life hard with e10s on Dev Edition before we do a large test like this?
Nominating for tracking for 44. Ritu, from discussion in email, this may now be aimed for 44 or 45.
status-firefox44: --- → affected
tracking-firefox44: --- → ?
Actually, scratch that, I think the work for the 44 experiment is in bug 1229104.
status-firefox44: affected → ---
tracking-firefox44: ? → ---
We can't do this until Fx45 at the earliest because of the Fx44 experiment. Also a good reminder from bsmedberg is that the criteria to kick splitting the population in beta is to be sure that the M8 issues are resolved, first.
Flags: needinfo?(elancaster)
Blocks: e10s-beta
[Tracking Requested - why for this release]:
e10s rollout to beta targeting users without addons in fx 45
tracking-firefox45: --- → ?
The current experiment is essentially doing this split. Just with partial population, which can be increased to all.
I expect it can be easily modified to add two branches(IDs) that highlight users without addons.
Possible problem would be if a second experiment were expected to run at same time.
(In reply to Jonathan Howard from comment #6)
> The current experiment is essentially doing this split. Just with partial
> population, which can be increased to all.
> I expect it can be easily modified to add two branches(IDs) that highlight
> users without addons.
> Possible problem would be if a second experiment were expected to run at
> same time.

Jonathan, what are the next steps here?
Flags: needinfo?(jonathan)
I'm just an outside observer, so my actions are ad hoc. Only read half the planning and change of plans.

Input for release engineers probably a good idea. I know a11y will be without e10s and that proportion running this code path is more that a trivial percent. Right now the experiment gives the 50/50 split to population not running a11y so relative crash rate is easy to observe but this does not so easily map directly to overall crash rate taking account of ADI. Expect experiment branches could be maintained to keep overall 50/50 split. Crash stats search can then pick out the correct branches.
There is also non crash telemetry which AFAIK hasn't yet been looked at for the experiment; (since aurora 33.)

Vladan is possibly best to give-input/determine if experiment is suited to be extended to perform full split. (beta 45?)
Flags: needinfo?(jonathan) → needinfo?(vladan.bugzilla)
Brad: let's discuss the plans for Beta 45 at the e10s meeting
Flags: needinfo?(vladan.bugzilla)
Depends on: 1232274, 1234675
Summary: Split the beta population in two for e10s testing → Beta45 e10s experiment: split the beta population in two for e10s testing
Vladan said he'd be working on this one and will ask me to review
Assignee: felipc → vladan.bugzilla
Status: NEW → ASSIGNED
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → DUPLICATE
You need to log in before you can comment on or make changes to this bug.