1496181 - [Shield] Opt-out Study: Activity Stream Contextual Feature Recommender - 63 Release Channel

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Tim Spurway [:tspurway]]

This study has exactly the same parameters and expectations of the beta 63 study (#1491379) it is derived from, except the sizing should be something more reasonable for the release channel.

Comment 1

[Tim Spurway [:tspurway]]

:matt_g how should we size this study (does 1% of en-US seem reasonable/powerful enough?)

The slug id for this study should be *prefixed with*:  prefflip-activity-stream-cfr-release-63

Comment 2

[Matt Grimes [:Matt_G]]

That seems reasonable, but bmiroglio owns the final population size. I'll let him comment.

Comment 3

[Tim Spurway [:tspurway]]

:ilana r? for science
:k88hudson r? for peer
:mcoman r? for QA
:jcristau r? for relman 
:osunick r? for VP

Comment 4

[Ilana]

Science: R+

Comment 5

[Ben Miroglio [:bmiroglio]]

1% is more than powerful enough--with the following note:

To calculate retention we'll need ~40k users per branch *minimum*. A 1% sample will likely hit that target in a day or two, so we should ensure to stop continually enrolling users into the study after 3-4 days, to give us some padding as we'll need to filter out some users before analysis (as is standard with all experiments).

Comment 6

[Kate Hudson :k88hudson]

Peer: R+

Comment 7

[MarniePW [:marnie]]

[Tracking Requested - why for this release]:

Comment 8

[Julien Cristau [:jcristau]]

redirect to Pascal as he's looking after 63

Comment 9

[Nick Nguyen [:osunick]]

Approved.

Comment 10

[Pascal Chevrel:pascalc]

This has relman approval.

Comment 11

[Marius Coman [:mcoman], Ecosystem QA]

Activity Stream Contextual Feature Recommender - 63 Release Channel

Targeted: Firefox Release 63

We have finished testing the Activity Stream Contextual Feature Recommender experiment.

QA’s recommendation:  RED - DON'T SHIP

Reasoning:

- We have found a blocker (Bug 1501656)  that affects the “Experiment 2” branch. The recommendation for the “Facebook” and “Reddit” websites will be displayed only two times per add-on instead of three.

Testing Summary:

- Full Functional test suite: https://goo.gl/G3dUFj;

Tested Platforms:

- Windows 10 x64

- Mac 10.13.3

- Arch Linux 4.16.6 x64

Tested Firefox versions:

- Firefox 63

Comment 12

[Tim Spurway [:tspurway]]

Thanks, Marius.

We have identified the bug and have a patch in the works, which we intend to ask for uplift into Beta 64.  

:mcoman, because the study is split into two experiment branches (one with freq_cap=1 and one with freq_cap=3), I propose we  run the study with a single experiment branch where freq_cap=1.  Because the undesired behaviour happens for freq_cap values > 1, could we get an r+ for an experiment with the single branch?

Comment 13

[Marius Coman [:mcoman], Ecosystem QA]

Activity Stream Contextual Feature Recommender - 63 Release Channel
Targeted: Firefox Release 63

We have finished testing the Activity Stream Contextual Feature Recommender experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:
- Considering the fact that the “Experiment 2” branch, which is affected by Bug 1501656, will not be part of this shield study, and the fact that the user enrollment will be performed through “Normandy” and it cannot be tested by us, the study is good to go from the QA point of view.

Testing Summary:
- Full Functional test suite: https://goo.gl/G3dUFj;

Tested Platforms:
- Windows 10 x64
- Mac 10.13.3
- Arch Linux 4.16.6 x64

Tested Firefox versions:
- Firefox 63

Comment 14

[Tim Spurway [:tspurway]]

We took a look at the telemetry, and the pings for Reddit and Facebook addons are being double counted for the recommendation.  Let's put this study on hold for now.

Comment 15

[Tim Spurway [:tspurway]]

We have landed a patch to fix and have requested uplift to beta 64 and ride-along to release 63 if possible (bug #1501656)

Comment 16

[Tim Spurway [:tspurway]]

r? :mcoman.

63.0.1 Release includes the fix for Bug #1501656, and so we should now be able to launch the experiment as originally specified with both cohorts (freq caps 1 and 3).  Could we get a quick re-test of the original?

Comment 17

[Marius Coman [:mcoman], Ecosystem QA]

Hi Tim, first of all, in order to test this study on a short period of time we would have to run our test suites only on two OS's, more precisely on Windows 10 x64 and Mac 10.13.3 and even in this case we'll need at least two days. That means we can test the CFR shield study, send the sign-off and launch on Tuesday morning (6th November 2018) your time. 

Also, if this is ok for you, could you please send a reply to the original PI e-mail in order to announce the change of plans.

Comment 18

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

(In reply to Pascal Chevrel:pascalc from comment #10)
> This has relman approval.

Hi Pascal, this time NI for review for running this study on release 63 in the coming week. I am told QA is in progress.

Comment 19

[Pascal Chevrel:pascalc]

Provided QA give their green light after the re-test of the study for tomorrow, this is a+ for relman.

Comment 20

[Marius Coman [:mcoman], Ecosystem QA]

Hi Pascal, from Tim's email it seems that the launch date of the "CFR" shield study was postponed to November 12, in order to provide enough time for the QA to test the study.

Comment 21

[Tim Spurway [:tspurway]]

Hey folks, I wanted to give QA enough time to properly re-test the feature, and a Nov 12 launch date still gives us lots of time to gather data from the study.  If, on the other hand, the QA team is done sooner, then let's by all means launch the study sooner.

Comment 22

[Marius Coman [:mcoman], Ecosystem QA]

Activity Stream Contextual Feature Recommender - 63 Release Channel

Targeted: Firefox Release 63

We have finished testing the Activity Stream Contextual Feature Recommender shield study experiment.

During testing we have found the following new issues:

    Bug 1505450 - The "CFR" recommendation is wrongly displayed on "Facebook" and "YouTube" websites if a similar add-on is installed and the website is focused when the is browser restarted

    Bug 1506135 - The CFR recommandation is wrongly displayed on an untargeted tab if you quickly switch to it from a targeted one


QA’s recommendation: YELLOW - SHIP CONDITIONALLY

Reasoning:

- We do not consider the two issues as blockers for the Study. However, we recommend fixing them as quickly as possible because they could affect a small percent of users.

Testing Summary:

- Full Functional test suite: https://goo.gl/G3dUFj;

Tested Platforms:

- Windows 10 x64;

- Mac 10.13.3

- Arch Linux 4.16.6 x64

Tested Firefox versions:

- Firefox release 63.0.1

Comment 23

[MarniePW [:marnie]]

Pascal, QA gave this a Yellow. How do you feel about shipping this study on Monday?

Comment 24

[Pascal Chevrel:pascalc]

(In reply to Marnie Pasciuto-Wood [:marnie] from comment #23)
> Pascal, QA gave this a Yellow. How do you feel about shipping this study on
> Monday?

Yes it is ok for me.

Comment 25

[Matt Grimes [:Matt_G]]

This study is now live with the following details:

* 1% of en_US Release 63+
* Slug = pref-flip-activity-stream-cfr-release-63 (note the additional "-" to conform with the standard. This is different than what tspurway posted above)
* User preference branch
* Branches:
[
  {
    "ratio": 1,
    "slug": "Control",
    "value": "[]"
  },
  {
    "ratio": 1,
    "slug": "Cohort1",
    "value": "[{\"id\":\"cfr\", \"cohort\": \"one_per_day\", \"frequency\": {\"custom\": [{\"period\": \"daily\", \"cap\": 1}]}, \"type\":\"local\",\"localProvider\":\"CFRMessageProvider\",\"enabled\":true}]"
  },
  {
    "ratio": 1,
    "slug": "Cohort2",
    "value": "[{\"id\":\"cfr\", \"cohort\": \"three_per_day\", \"frequency\": {\"custom\": [{\"period\": \"daily\", \"cap\": 1}]}, \"type\":\"local\",\"localProvider\":\"CFRMessageProvider\",\"enabled\":true}]"
  }
]

Comment 26

[Robert (rrayborn, he/him)]

We've closed this study.  Thanks

Comment 27

[Pascal Chevrel:pascalc]

Are you keeping this bug open to post results of the study later?

Comment 28

[:shell escalante]

we are adding an area in Experimenter for results. Something is also landing to auto-close the "experimenter" created bugs when the experiments are completed.