Last Comment Bug 1253669 - Run an e10s A/B experiment - Beta 46 - APZ off (third experiment)
: Run an e10s A/B experiment - Beta 46 - APZ off (third experiment)
Status: RESOLVED FIXED
:
Product: Firefox
Classification: Client Software
Component: General (show other bugs)
: 45 Branch
: Unspecified Unspecified
-- normal (vote)
: ---
Assigned To: :Felipe Gomes (needinfo me!)
:
:
Mentors:
Depends on: 1253667 1255094
Blocks: 1260769
  Show dependency treegraph
 
Reported: 2016-03-04 10:22 PST by :Felipe Gomes (needinfo me!)
Modified: 2016-03-30 08:29 PDT (History)
14 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
m9+


Attachments
patch (6.52 KB, patch)
2016-03-08 08:41 PST, :Felipe Gomes (needinfo me!)
dtownsend: review+
Details | Diff | Splinter Review
new experiment.xpi, unsigned (2.05 KB, application/x-xpinstall)
2016-03-08 14:15 PST, :Felipe Gomes (needinfo me!)
jthomas: feedback+
Details
new experiment.xpi, signed (5.93 KB, application/x-xpinstall)
2016-03-08 14:29 PST, Jason Thomas [:jason]
no flags Details

Description User image :Felipe Gomes (needinfo me!) 2016-03-04 10:22:47 PST
First experiment on beta46, with apz off (bug 1253667)

The goal is to ship this to 100% of the population for two weeks, given that only 20% of users will end up with e10s activated (due to the test/control splitting necessary for statistical significance). And the previous experiment have well demonstrated that the experiment doesn't impact the other 80% of users in any way.
Comment 1 User image Jonathan Howard 2016-03-04 13:03:06 PST
Note the beta45 experiments had minor inconvenience of accessibility adding ~10% extra to control population.
Comment 2 User image :Felipe Gomes (needinfo me!) 2016-03-08 08:40:02 PST
(In reply to :Felipe Gomes (needinfo me!) from comment #0)
> First experiment on beta46, with apz off (bug 1253667)
> 
> The goal is to ship this to 100% of the population for two weeks, given that
> only 20% of users will end up with e10s activated (due to the test/control
> splitting necessary for statistical significance). And the previous
> experiment have well demonstrated that the experiment doesn't impact the
> other 80% of users in any way.

Turns out we can't do that because the crash rates dashboard separates users who are part of experiments, regardless of the way their grouping works. So we'll continue with 50%
Comment 3 User image :Felipe Gomes (needinfo me!) 2016-03-08 08:41:40 PST
Created attachment 8727922 [details] [diff] [review]
patch

Basically just manifest changes..  Only change in the code is that we no longer need to manually set extensions.e10sBlocksEnabling because that is now in the tree for 46 (bug 1250744).
Comment 4 User image Robert Kaiser 2016-03-08 10:23:30 PST
(In reply to :Felipe Gomes (needinfo me!) from comment #2)
> Turns out we can't do that because the crash rates dashboard separates users
> who are part of experiments, regardless of the way their grouping works. So
> we'll continue with 50%

I think it would be better to fix those dashboards to actually use people with e10s off or at least people including the control group instead of making this limit our population in that way.
Comment 5 User image Erin Lancaster [:elan] 2016-03-08 10:30:40 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #4)
> (In reply to :Felipe Gomes (needinfo me!) from comment #2)
> > Turns out we can't do that because the crash rates dashboard separates users
> > who are part of experiments, regardless of the way their grouping works. So
> > we'll continue with 50%
> 
> I think it would be better to fix those dashboards to actually use people
> with e10s off or at least people including the control group instead of
> making this limit our population in that way.

+1 :Felipe, does this negatively impact the quality of data we will be obtaining? I feel like this time is precious and we should keep e10s exposure as high as possible. Can we not fix the dashboard, instead?
Comment 6 User image Jeff Griffiths (:canuckistani) (:⚡︎) 2016-03-08 10:32:53 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #4)
> (In reply to :Felipe Gomes (needinfo me!) from comment #2)
> > Turns out we can't do that because the crash rates dashboard separates users
> > who are part of experiments, regardless of the way their grouping works. So
> > we'll continue with 50%
> 
> I think it would be better to fix those dashboards to actually use people
> with e10s off or at least people including the control group instead of
> making this limit our population in that way.

Sounds good to me - can we do this quickly? Who does it?
Comment 7 User image Benjamin Smedberg 2016-03-08 10:56:11 PST
My understanding of Friday's meeting is that we were going to run an experiment on 50% of the population. I don't understand why we need more: we have more than enough users to validate stats. If we're going split 100% of the population we might as well ship the progressive rollout system addon. An experiment sucks for that because we'd be blocking all other experiments.
Comment 8 User image Jonathan Howard 2016-03-08 11:29:11 PST
Example of more is better;
Overall fewer OOM crashes. (including content.)
Currently browser process only, so not great. Until bug 1236108.
2G very good. ~10% crashes
1.5G samples size seems low. seems ~50%
1G again ~50%
0.5G ~100%

(10% extra users in control sample than experiment)
https://crash-stats.mozilla.com/api/SuperSearch/?product=Firefox&signature=%3Dmoz_abort+|+pages_commit&signature=~OOM&date=%3E%3D2016-02-12&date=%3C2016-02-25&ActiveExperimentBranch=~no-addons&ActiveExperiment=~e10s-beta45-withoutaddons&_facets=dom_ipc_enabled&_facets=ActiveExperimentBranch&_columns=build_id&_columns=process_type&_columns=dom_ipc_enabled&_columns=total_physical_memory&_results_number=0&_histogram.total_physical_memory=ActiveExperimentBranch&_histogram_interval.total_physical_memory=536870912

System addon AFAIK isn't reporting the cohort in crashes, Non insignificant number of beta e10s crashes are by users outside of experiment but stable comparison population is far better for comparison.
Comment 9 User image Robert Kaiser 2016-03-08 11:44:52 PST
(In reply to Jeff Griffiths (:canuckistani) (:⚡︎) from comment #6)
> Sounds good to me - can we do this quickly? Who does it?

I don't even know what dashboards you tzalk about, but if it's bsmedberg's ones, it's him that needs to take care of that.
Comment 10 User image :Felipe Gomes (needinfo me!) 2016-03-08 14:15:09 PST
Created attachment 8728104 [details]
new experiment.xpi, unsigned

Hi Jason! Can you sign this xpi for me?
Comment 11 User image Jason Thomas [:jason] 2016-03-08 14:29:22 PST
Created attachment 8728108 [details]
new experiment.xpi, signed

Please see attached.
Comment 12 User image :Felipe Gomes (needinfo me!) 2016-03-09 08:44:41 PST
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/3668dff24e33

I'll send an e-mail to release-drivers and file a new bug for deployment.
Comment 13 User image :Felipe Gomes (needinfo me!) 2016-03-09 10:40:37 PST
Liz signed off, I updated the release_tag:
https://hg.mozilla.org/webtools/telemetry-experiment-server/rev/aa41f7d5582b

And filed bug 1255094 for deployment.

Note You need to log in before you can comment on or make changes to this bug.