1253669 - Run an e10s A/B experiment - Beta 46 - APZ off (third experiment)

Status: RESOLVED FIXED

(Firefox :: General, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

First experiment on beta46, with apz off (bug 1253667)

The goal is to ship this to 100% of the population for two weeks, given that only 20% of users will end up with e10s activated (due to the test/control splitting necessary for statistical significance). And the previous experiment have well demonstrated that the experiment doesn't impact the other 80% of users in any way.

Comment 1

[Jonathan Howard]

Note the beta45 experiments had minor inconvenience of accessibility adding ~10% extra to control population.

Comment 2

[:Felipe Gomes (needinfo for replies!)]

(In reply to :Felipe Gomes (needinfo me!) from comment #0)
> First experiment on beta46, with apz off (bug 1253667)
> 
> The goal is to ship this to 100% of the population for two weeks, given that
> only 20% of users will end up with e10s activated (due to the test/control
> splitting necessary for statistical significance). And the previous
> experiment have well demonstrated that the experiment doesn't impact the
> other 80% of users in any way.

Turns out we can't do that because the crash rates dashboard separates users who are part of experiments, regardless of the way their grouping works. So we'll continue with 50%

Comment 3

[:Felipe Gomes (needinfo for replies!)]

Attachment: patch

Basically just manifest changes..  Only change in the code is that we no longer need to manually set extensions.e10sBlocksEnabling because that is now in the tree for 46 (bug 1250744).

Comment 4

[Robert Kaiser]

(In reply to :Felipe Gomes (needinfo me!) from comment #2)
> Turns out we can't do that because the crash rates dashboard separates users
> who are part of experiments, regardless of the way their grouping works. So
> we'll continue with 50%

I think it would be better to fix those dashboards to actually use people with e10s off or at least people including the control group instead of making this limit our population in that way.

Comment 5

[Erin Lancaster [:elan]]

(In reply to Robert Kaiser (:[e-mail]) from comment #4)
> (In reply to :Felipe Gomes (needinfo me!) from comment #2)
> > Turns out we can't do that because the crash rates dashboard separates users
> > who are part of experiments, regardless of the way their grouping works. So
> > we'll continue with 50%
> 
> I think it would be better to fix those dashboards to actually use people
> with e10s off or at least people including the control group instead of
> making this limit our population in that way.

+1 :Felipe, does this negatively impact the quality of data we will be obtaining? I feel like this time is precious and we should keep e10s exposure as high as possible. Can we not fix the dashboard, instead?

Comment 6

[Jeff Griffiths (:canuckistani) (:⚡︎)]

(In reply to Robert Kaiser (:[e-mail]) from comment #4)
> (In reply to :Felipe Gomes (needinfo me!) from comment #2)
> > Turns out we can't do that because the crash rates dashboard separates users
> > who are part of experiments, regardless of the way their grouping works. So
> > we'll continue with 50%
> 
> I think it would be better to fix those dashboards to actually use people
> with e10s off or at least people including the control group instead of
> making this limit our population in that way.

Sounds good to me - can we do this quickly? Who does it?

Comment 7

[Benjamin Smedberg]

My understanding of Friday's meeting is that we were going to run an experiment on 50% of the population. I don't understand why we need more: we have more than enough users to validate stats. If we're going split 100% of the population we might as well ship the progressive rollout system addon. An experiment sucks for that because we'd be blocking all other experiments.

Comment 8

[Jonathan Howard]

Example of more is better;
Overall fewer OOM crashes. (including content.)
Currently browser process only, so not great. Until bug 1236108.
2G very good. ~10% crashes
1.5G samples size seems low. seems ~50%
1G again ~50%
0.5G ~100%

(10% extra users in control sample than experiment)
https://crash-stats.mozilla.com/api/SuperSearch/?product=Firefox&signature=%3Dmoz_abort+|+pages_commit&signature=~OOM&date=%3E%3D2016-02-12&date=%3C2016-02-25&ActiveExperimentBranch=~no-addons&ActiveExperiment=~e10s-beta45-withoutaddons&_facets=dom_ipc_enabled&_facets=ActiveExperimentBranch&_columns=build_id&_columns=process_type&_columns=dom_ipc_enabled&_columns=total_physical_memory&_results_number=0&_histogram.total_physical_memory=ActiveExperimentBranch&_histogram_interval.total_physical_memory=536870912

System addon AFAIK isn't reporting the cohort in crashes, Non insignificant number of beta e10s crashes are by users outside of experiment but stable comparison population is far better for comparison.

Comment 9

[Robert Kaiser]

(In reply to Jeff Griffiths (:canuckistani) (:⚡︎) from comment #6)
> Sounds good to me - can we do this quickly? Who does it?

I don't even know what dashboards you tzalk about, but if it's bsmedberg's ones, it's him that needs to take care of that.

Comment 10

[:Felipe Gomes (needinfo for replies!)]

Attachment: new experiment.xpi, unsigned

Hi Jason! Can you sign this xpi for me?

Comment 11

[Jason Thomas [:jason]]

Attachment: new experiment.xpi, signed

Please see attached.

Comment 12

[:Felipe Gomes (needinfo for replies!)]

https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/3668dff24e33

I'll send an e-mail to release-drivers and file a new bug for deployment.

Comment 13

[:Felipe Gomes (needinfo for replies!)]

Liz signed off, I updated the release_tag:
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/aa41f7d5582b

And filed bug 1255094 for deployment.