Run an e10s A/B experiment - Beta 46 - APZ off (third experiment)

RESOLVED FIXED

Status

()

Firefox
General
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: Felipe, Assigned: Felipe)

Tracking

(Blocks: 1 bug)

45 Branch
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(e10sm9+)

Details

Attachments

(3 attachments)

(Assignee)

Description

a year ago
First experiment on beta46, with apz off (bug 1253667)

The goal is to ship this to 100% of the population for two weeks, given that only 20% of users will end up with e10s activated (due to the test/control splitting necessary for statistical significance). And the previous experiment have well demonstrated that the experiment doesn't impact the other 80% of users in any way.

Comment 1

a year ago
Note the beta45 experiments had minor inconvenience of accessibility adding ~10% extra to control population.
(Assignee)

Updated

a year ago
Summary: Run an e10s A/B experiment - Beta 46 - APZ off (first experiment) → Run an e10s A/B experiment - Beta 46 - APZ off (third experiment)
(Assignee)

Comment 2

a year ago
(In reply to :Felipe Gomes (needinfo me!) from comment #0)
> First experiment on beta46, with apz off (bug 1253667)
> 
> The goal is to ship this to 100% of the population for two weeks, given that
> only 20% of users will end up with e10s activated (due to the test/control
> splitting necessary for statistical significance). And the previous
> experiment have well demonstrated that the experiment doesn't impact the
> other 80% of users in any way.

Turns out we can't do that because the crash rates dashboard separates users who are part of experiments, regardless of the way their grouping works. So we'll continue with 50%
(Assignee)

Comment 3

a year ago
Created attachment 8727922 [details] [diff] [review]
patch

Basically just manifest changes..  Only change in the code is that we no longer need to manually set extensions.e10sBlocksEnabling because that is now in the tree for 46 (bug 1250744).
Attachment #8727922 - Flags: review?(dtownsend)
Attachment #8727922 - Flags: review?(dtownsend) → review+

Comment 4

a year ago
(In reply to :Felipe Gomes (needinfo me!) from comment #2)
> Turns out we can't do that because the crash rates dashboard separates users
> who are part of experiments, regardless of the way their grouping works. So
> we'll continue with 50%

I think it would be better to fix those dashboards to actually use people with e10s off or at least people including the control group instead of making this limit our population in that way.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #4)
> (In reply to :Felipe Gomes (needinfo me!) from comment #2)
> > Turns out we can't do that because the crash rates dashboard separates users
> > who are part of experiments, regardless of the way their grouping works. So
> > we'll continue with 50%
> 
> I think it would be better to fix those dashboards to actually use people
> with e10s off or at least people including the control group instead of
> making this limit our population in that way.

+1 :Felipe, does this negatively impact the quality of data we will be obtaining? I feel like this time is precious and we should keep e10s exposure as high as possible. Can we not fix the dashboard, instead?
Flags: needinfo?(felipc)
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #4)
> (In reply to :Felipe Gomes (needinfo me!) from comment #2)
> > Turns out we can't do that because the crash rates dashboard separates users
> > who are part of experiments, regardless of the way their grouping works. So
> > we'll continue with 50%
> 
> I think it would be better to fix those dashboards to actually use people
> with e10s off or at least people including the control group instead of
> making this limit our population in that way.

Sounds good to me - can we do this quickly? Who does it?
Flags: needinfo?(kairo)
My understanding of Friday's meeting is that we were going to run an experiment on 50% of the population. I don't understand why we need more: we have more than enough users to validate stats. If we're going split 100% of the population we might as well ship the progressive rollout system addon. An experiment sucks for that because we'd be blocking all other experiments.

Comment 8

a year ago
Example of more is better;
Overall fewer OOM crashes. (including content.)
Currently browser process only, so not great. Until bug 1236108.
2G very good. ~10% crashes
1.5G samples size seems low. seems ~50%
1G again ~50%
0.5G ~100%

(10% extra users in control sample than experiment)
https://crash-stats.mozilla.com/api/SuperSearch/?product=Firefox&signature=%3Dmoz_abort+|+pages_commit&signature=~OOM&date=%3E%3D2016-02-12&date=%3C2016-02-25&ActiveExperimentBranch=~no-addons&ActiveExperiment=~e10s-beta45-withoutaddons&_facets=dom_ipc_enabled&_facets=ActiveExperimentBranch&_columns=build_id&_columns=process_type&_columns=dom_ipc_enabled&_columns=total_physical_memory&_results_number=0&_histogram.total_physical_memory=ActiveExperimentBranch&_histogram_interval.total_physical_memory=536870912

System addon AFAIK isn't reporting the cohort in crashes, Non insignificant number of beta e10s crashes are by users outside of experiment but stable comparison population is far better for comparison.

Comment 9

a year ago
(In reply to Jeff Griffiths (:canuckistani) (:⚡︎) from comment #6)
> Sounds good to me - can we do this quickly? Who does it?

I don't even know what dashboards you tzalk about, but if it's bsmedberg's ones, it's him that needs to take care of that.
Flags: needinfo?(kairo)
(Assignee)

Updated

a year ago
Flags: needinfo?(felipc)
(Assignee)

Comment 10

a year ago
Created attachment 8728104 [details]
new experiment.xpi, unsigned

Hi Jason! Can you sign this xpi for me?
Attachment #8728104 - Flags: feedback?(jthomas)
Attachment #8728104 - Flags: feedback?(jthomas) → feedback+
Created attachment 8728108 [details]
new experiment.xpi, signed

Please see attached.

Updated

a year ago
tracking-e10s: --- → m9+
(Assignee)

Comment 12

a year ago
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/3668dff24e33

I'll send an e-mail to release-drivers and file a new bug for deployment.
Status: ASSIGNED → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
(Assignee)

Updated

a year ago
Depends on: 1255094
(Assignee)

Comment 13

a year ago
Liz signed off, I updated the release_tag:
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/aa41f7d5582b

And filed bug 1255094 for deployment.
Blocks: 1260769
You need to log in before you can comment on or make changes to this bug.