1380005 - [Shield] Opt-out Study Tab Center

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Peter Dolanjski [:pdol]]

Measurements
To quantify success, we need to track the following metrics:
- Sentiment measures
-- Full questionnaire
- Heavy user traits
-- Page views
-- Average number of tabs
-- usage hours/session hours
- Additional measurements (useful for further product/feature development, qualifying results etc.)
-- Number of searches (Yahoo etc…)
-- Number of searches for tabs in the side bar
-- Number of mode switches (tabs on top mode vs. side tabs mode)
-- Frequency of mode switches
-- Time spent in each mode
-- Number of tabs in each mode
-- Number of tabs on switch
--- The goal here is to know what tab number causes users to switch from one mode into another. We need separate measures for the two switching »directions«
-- Progress/dropoff rate for onboarding
--- Outlined in more detail in the onboarding spec
-- Click through rate when the hint is shown (when a user is in tabs-on-top mode, we can show a hint to switch to tab center)

Segmentation
- Time spent in side tabs mode
- Influencers

Hypotheses (formal version)
Primary
- Sentiment measures of A > Sentiment measures of B (by 2%+)
- Number of pageviews of C > Number of pageviews of D (by 1%+)
- Retention of E >= than retention of F (by 1%+)

Secondary
- Total usage hours of A, C > total usage hours of B, D, respectively
- Number of searches in A, C, E >= Number of searches in B, D, F, respectively

Note: Any one of these can be considered success so long as it is not accompanied by a drop in the key metrics for the other success criteria. (Eg. If sentiment of A goes up relative to B, but retention of E gets worse than retention of F, that wouldn’t necessarily be considered success)

Experiment duration
We believe that we can see results within one month from enrollment. We are targeting mid-August 2017 for the launch.

Comment 1

[Peter Dolanjski [:pdol]]

The telemetry for the experiment can be found here:
https://github.com/bwinton/TabCenter/blob/master/docs/metrics.md

Comment 2

[Peter Dolanjski [:pdol]]

Saptarshi, can you validate the permissibly to collect the telemetry data linked above?

Comment 3

[Erica Wright [:ewright]]

Attachment: tab-center.xpi

Comment 4

[Erica Wright [:ewright]]

Attachment: tab-center.xpi

Comment 5

[Erica Wright [:ewright]]

Attachment: tab-center.xpi

Comment 6

[Matt Grimes [:Matt_G]]

We plan on shipping this in Release 56. Can we get the add-on signed please?

Comment 7

[Osmose [:osmose, :mkelly]]

Since this is going to release, we need sign-off on feature testing from a QE team. If ya'll already have it, can you link to it from this bug?

Comment 8

[Carmen Fat [:cfat] - Ecosystem QA]

Hi guys,

We have ran our set of manual tests on Firefox 56 using the provided build and we uncovered a couple of issues. All of them were addressed and are verified as fixed (for more details about the testing we have done, please access the following link: https://testrail.stage.mozaws.net/index.php?/plans/view/6374).

We found another issue today (https://github.com/bwinton/TabCenter/issues/1127), but since this isn't something major, from manual QA perspective we consider that we are good to go.

Comment 9

[Michael Cooper [:mythmon]]

Attachment: tab-center-1.37.0-signed.xpi

I talked with Erica on Slack, and she clarified that the XPI uploaded in comment 5 includes the fixes mentioned in comment 8. Given that, I've signed it, and uploaded it Shield's admin interface. I've also attached it here.

Comment 10

[Liz Henry (:lizzard) (relman/hg->git project)]

Sounds good, please go ahead with the rollout for release 56.

Comment 11

[Robert (rrayborn, he/him)]

We're live.

Comment 12

[Josh Gaunt [:jgaunt]]

Deployed a CEP filter to track DAU in this study:

https://pipeline-cep.prod.mozaws.net/dashboard_output/analysis.jgaunt.tabcenter_estimates.tabcentertest1_count.json

Comment 13

[Matt Grimes [:Matt_G]]

@ilana Now that the study has ended, can we get a quick recap of the results and then have this bug closed? A couple sentences is fine.

Comment 14

[Ilana]

Transferred to jgaunt

Comment 15

[Josh Gaunt [:jgaunt]]

Completing a thorough between-subjects analysis depends on https://bugzilla.mozilla.org/show_bug.cgi?id=1445774, within-subject results are currently available but we don't consider that the most valid inference possible

Comment 16

[Josh Gaunt [:jgaunt]]

Tab Center results have been finalized: https://gist.github.com/jtg567/6e029abc2f4e61b7e5a6b319e8d34d2f

Summary of significant differences:

    Variant showed a small increase in unique_domains_count (3.9) vs. a small decrease in Control (-1.4). Significant at p < .05

    Variant showed a greater decrease in active_hours (-4.2) vs. Control (-2.8). Significant at p < .05

So, the experimental/variant group visited more unique domains and had shorter sessions than control; and those differences are statistically reliable. The remaining metrics showed no such reliable differences.

Comment 17

[Matt Grimes [:Matt_G]]

I'm going to close this bug now that the analysis has been posted. Any followup requests can be sent to jgaunt directly.