1223895 - Deploy a Heartbeat survey on e10s A/B experiment participants

Status: RESOLVED WONTFIX

(Toolkit :: Telemetry, defect)

Description

[Vladan Djeric (:vladan)]

I would like to deploy a Heartbeat survey asking e10s & non-e10s users to rate their satisfaction with Firefox performance and analyze their results.

The data from this survey could also be used to develop a "session responsiveness" scoring formula.

Comment 1

[Gregg Lind (Fx Strategy and Insights - Shield - Heartbeat )]

As keeper of surveyGizmo, I approve this idea.

Design proposal for study.

Claims:

1.  e10s has several mechanisms of harming and aiding users.
2.  For NEW users, faster vidyo performance should be GOOD.  (all upside).
3.  For EXISTING users, e10s can break addons, degrading experience.  

I would like to see it as part of a FORCED SWITCH (e10s on/off randomly assigned) for a subset of NEW and EXISTING firefox users. 

Starting from users with e10s OFF, 4 cohorts:
- new users, e10s forces on.
- new users, e10s forced off.
- existing users (with addons?), e10s forced on.
- existing users (with addons?), e10s forced off.


I would like to see these outcomes tracked (by cohort):

- retention
- total usage 

Covariates to track:
- channel
- addons

Comment 2

[Matt Grimes [:Matt_G]]

Adding Jeff since we just spoke about this yesterday.

Comment 3

[Erin Lancaster [:elan]]

Confirming we'd like to do this for Firefox 46 Beta Experiment (Phase 2, March 21. If we could do it sooner, that would be even better). We should compare the heartbeat score of e10s and non-e10s cohorts. I'll follow up on getting Bug 1193535 uplifted.

Comment 4

[Matthew N. [:MattN]]

(In reply to Erin Lancaster [:elan] from comment #3)
> I'll follow up on getting Bug 1193535 uplifted.

It's should already be in 46.

Comment 5

[Ilana]

Update: We are indeed seeing data coming in (we confused merge date and deploy date for our first check). Volume by date, for any interested parties:

[(u'20160316', 2689),
 (u'20160315', 2751),
 (u'20160314', 2167),
 (u'20160313', 1309),
 (u'20160312', 637),
 (u'20160311', 10)]

Comment 6

[Erin Lancaster [:elan]]

:Ilana how are ratings looking? Is it possible to do any comparisons at this point? Thank you.

Comment 7

[Ilana]

Felipe, 
I need a little more info to make sure I'm doing a valid comparison across populations. Here's what I've understood from my conversations with elan so far, and some points that need clarification:


- Only consider users who were ELIGIBLE for experiment 3, which had two rollout phases. Compare ratings from the e10s enabled group to the same population that was not exposed to e10s.

Eligibility for experiment 3 is:
- Beta 46
- rating submitted on 3/23 or later
- no addons (I assume besides system addons - pocket and hello)
- no r to l languages (is this in a pref?)
- no accessibility issues (is this in a pref?)

Implementation:
- In phase 1, apz was disabled, and in phase 2, it was enabled. Were these done at the same or different times?
- Phase 1 subjects can be identified by having an xpi - what is it called?
- Phase 2 subjects can be identified by having the [e-mail] addon.
- 20% of eligible users should be in either phase 1 or phase 2

Additionally:
- participants in experiments 1 and 2 should be discarded. 
          --> Can we can identify the users who saw exps 1 and 2 by a flipped e10s pref? We want to remove them from consideration. If so, which pref is this? If not, how can we identify these users?

If any of this is incorrect, please let me know, and let me know if you can answer the questions above.

Comment 8

[:Felipe Gomes (needinfo for replies!)]

Ok, this is a bit complicated.. Post-experiment identification of subjects was never a requirement, so it will be a bit hacky to figure that out. But I think it can work out. Can/does heartbeat run code on the local machine to decide this? That would be the only way to do it, I think.

Let's go by parts.

(In reply to Ilana from comment #7)
> Eligibility for experiment 3 is:
> - Beta 46
> - rating submitted on 3/23 or later
> - no addons (I assume besides system addons - pocket and hello)
> - no r to l languages (is this in a pref?)
> - no accessibility issues (is this in a pref?)

Not sure what you mean by "rating submitted on 3/23 or later".  The first phase ran from Mar 9 to Mar 21, and second phase from Mar 23 - Apr 4.

No addons/no rtl/no accessibility is correct, but you don't need to directly check them as the splitting into test/control already took care of only targetting eligible users.


> 
> Implementation:
> - In phase 1, apz was disabled, and in phase 2, it was enabled. Were these
> done at the same or different times?
> - Phase 1 subjects can be identified by having an xpi - what is it called?

It was an xpi but it got uninstalled at the end of the experiment.

> - Phase 2 subjects can be identified by having the [e-mail]
> addon.

Also an xpi but it was installed (and remains installed) for all users, not just eligible.

So the approach of identifying through xpis will not work. I have suggestions further below.

> Additionally:
> - participants in experiments 1 and 2 should be discarded. 
>           --> Can we can identify the users who saw exps 1 and 2 by a
> flipped e10s pref? We want to remove them from consideration. If so, which
> pref is this? If not, how can we identify these users?

I assume by "experiments 1 and 2" you mean the ones we ran in Beta 45? I don't think these users should be excluded, because they are the same target users of the experiments in beta 46. There will be a big intersection of users that would be excluded.


> 
> If any of this is incorrect, please let me know, and let me know if you can
> answer the questions above.

====== Phase 1 ======

In order to identify users from phase 1, we will need to consult the experiment system about it. Luckily it stores information on past experiments.

You'll need:

> function promisePhase1Cohort() {
>   let { Experiments } = Cu.import("resource:///modules/experiments/Experiments.jsm", {});
>   let exp = Experiments.instance();
>   return exp.getExperimentBranch("[e-mail]");
> }

This will return a promise that will evaluate to a string. Only consider the values of:
 - "control-no-addons": users who were part of the control group
 - "experiment-no-addons": users who were part of the test group

and ignore users with any other values. Note that null values might be returned.


===== Phase 2 =====

This one will be hackier.. The system add-on doesn't store information about the phase that ran, so we'll need to replicate the logic that it used to decide.

> function getPhase2Cohort() {
>   Cu.import("resource://gre/modules/Preferences.jsm");
> 
>   let uneligible = Preferences.get("browser.tabs.remote.autostart", false) ||
>                    Preferences.get("browser.tabs.remote.force-enable", false) ||
>                    Preferences.get("browser.tabs.remote.force-disable", false) ||
>                    (Services.appinfo.multiprocessBlockPolicy != 0) ||
>                    (Preferences.isSet("browser.tabs.remote.autostart.2") &&
>                     (Preferences.get("browser.tabs.remote.autostart.2") == false));
> 
>   let sample = Preferences.get("e10s.rollout.cohortSample", undefined);
> 
>   if (uneligible || (typeof(sample) != "number")) {
>     return "ignore";
>   }
> 
>   if (sample < 50) {
>     return "test";
>   } else {
>     return "control";
>   }
> }

This one is not a promise. Feel free to modify it as you see fit.  Please make sure to wrap both in a try {} catch, and to only run this on beta 46 users.

I don't know how heartbeat surveys work but I'd like to know more on how this code is gonna be used.

Comment 9

[:Felipe Gomes (needinfo for replies!)]

Hmm so maybe scratch that entire comment out. MattN has explained to me how heartbeat works.  I thought from the bug description that we were still going to deploy a survey and would need to identify users post-experiments.

My understanding now is that this survey already ran while the e10s A/B experiments where running, and the ratings are tied to telemetry pings, and now we want to analyse the ratings with the data that came in. Please let me know if this understanding is now correct.


Assuming this is true, here's how to find the users who were part of these experiments:

==== Phase 1 =====

Look for activeExperiment { id, branch } here: http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/docs/environment.rst?rev=b9ad1239e4b2#238

you should look for an id of: "[e-mail]"
The branch will tell you whether users were part of test/control groups. As mentioned above, only consider the users with these branches:
 - "control-no-addons": users who were part of the control group
 - "experiment-no-addons": users who were part of the test group

and ignore everyone else.

This should be enough to fully identify users who were part of this experiment, so you don't need date checks there. If you still need to know, this phase ran from Mar 9 - Mar 21.


==== Phase 2 =====

For this one, you'll need to look for e10sCohort here: http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/docs/environment.rst?rev=b9ad1239e4b2#46

Ignore users who don't have that set, as that was before the phase started. Here you need to consider only users with values "test" or "control".

In this case, you'll have to look at submissions only up to Apr 3, because the value will continue to exist afterwards, but ~100% of users will become "control" after that.

Comment 10

[Ilana]

This is exactly what I needed. Thank you for being so fast. And good job for including a control group, team!!!

I imagine that users who are in control and test are not conflated with users who may have e10s from another source (like one of the first two experiments, or manual pref switch), correct?

Comment 11

[:Felipe Gomes (needinfo for replies!)]

(In reply to Ilana from comment #10)
> I imagine that users who are in control and test are not conflated with
> users who may have e10s from another source (like one of the first two
> experiments, or manual pref switch), correct?

Right, manual pref switch users are on different groups not mentioned here. From the previous experiments, the test/control groups had their prefs cleared back to the default settings when the experiment finishes, so there's no effect of a past experiment on the distribution of the newer ones.

Comment 12

[Chris Peterson [:cpeterson]]

Erin is driving the Heartbeat survey.

Comment 13

[Ilana]

Do we know how long the experiment was deployed to users before HB survey was popped? Additionally, for those in the user-disabled, etc, category, do we have any way of knowing which branch they were in before the disable?

I assume the answer to both questions is no, but I want to check before presenting results.

Comment 14

[:Felipe Gomes (needinfo for replies!)]

(In reply to Ilana from comment #13)
> Do we know how long the experiment was deployed to users before HB survey
> was popped? Additionally, for those in the user-disabled, etc, category, do
> we have any way of knowing which branch they were in before the disable?

I don't know how Heartbeat pops its survey. The first experiment ran from Mar 9 to Mar 21, and the second from  Mar 22 to Apr 4. Perhaps heartbeat also stores the date that the user answered it, and you can compare from that?  Other than that, I don't know.

About the users in user-disabled branch, it's probably possible to do that if you do a longitudinal analysis looking for other telemetry pings from the same user, and check the previous values. But I think that's not worth it, as the number of users in those branches is small, and maybe they already started the experiment on those branches due to pre-existing prefs.

Comment 15

[Erin Lancaster [:elan]]

Are we good with HB working in Beta 48? Confirming we are extending the survey through this beta cycle.

Comment 16

[Matt Grimes [:Matt_G]]

We do have the standard heartbeat survey running on Beta right now. The general satisfaction score is running at all times. @ilana - How reusable is the analysis code you wrote for the first beta test? If I remember correctly it should be fairly easy to duplicate.

Comment 17

[Ilana]

(In reply to Matt Grimes [:Matt_G] from comment #16)
> How reusable is the analysis code you wrote for the first beta test?

100% reusable. Are we looking for the same metric - no significant loss of satisfaction? If so, do you want that as "overall," or channel-by-channel?

Comment 18

[Erin Lancaster [:elan]]

:bsmedberg: ^

Comment 19

[Benjamin Smedberg]

We want to continuously compare the "test" and "control" branches of the e10s experiment on beta, and starting with 48 release we want to compare the "test" and "control" branches on the release channel as well (separately by channel, since comparing channels is unlikely to be useful).

Comment 20

[Erin Lancaster [:elan]]

Ilana, can we obtain an updated report soon?

Comment 21

[Matt Grimes [:Matt_G]]

Just as an FYI, we are working on getting a dashboard up that will allow you to follow along in realtime as e10s moves through the trains and hits various milestones. I don't have a firm ETA, but Ilana should be able to provide adhoc reports until it is up.

Comment 22

[Ilana]

I can definitely run an updated report.

Are the experiment arms named the same thing? Also, are you interested in looking at all data aggregated since the last time the experiment was run (4/6) or only the most recent version? (The question that we addressed was "Is satisfaction the same for e10s users and non e10s users?" so that question can be answered for "all time," "recently," or by version).

Current report is at https://gist.github.com/ilanasegall/ec88e71e24e04d4c3f94ce2d6c58777d for reference.

Comment 23

[Erin Lancaster [:elan]]

By version would be good. We are wanting Firefox 48 at this point.

Comment 24

[:Felipe Gomes (needinfo for replies!)]

Hi, this time finding the branches is a bit different. The data to look for is no longer in environment.addons.activeExperiment, but rather at environment.settings.e10sCohort

That will contain the branch name, and you should compare users between the branches "test" and "control", and ignore any other name.

Comment 25

[Erin Lancaster [:elan]]

Hopefully, this is all the information you need. Thank you!

Comment 26

[Ilana]

Outstanding. And you are interested in Firefox 48 across only beta from 6/7 onwards?

Comment 27

[Erin Lancaster [:elan]]

If the sample size is big enough for the data to be meaningful, that's perfect. Beta 6 + sounds good.

Comment 28

[Ilana]

Updated report available at https://gist.github.com/ilanasegall/1dea80ff88647d8d98001bc46ee5f354

Bottom line: e10s cohort NOT statistically less satisfied, though we have a much smaller sample than we'd like.

note: original report can be seen here: https://gist.github.com/ilanasegall/717ac5fc9229b1c1548c1930fe644452/f356ef875a768ed4ac85c99af7fff768c8d69bdc#file-hb_explore-ipynb

Comment 29

[Alessio Placitelli [:Dexter]]

e10s shipped, this is no longer needed. Closing it off!