1373676 - [SHIELD] Study Validation Review for Containers Experiment

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[John Gruen [:jgruen]]

Hypothesis document is here:
https://docs.google.com/document/d/1vMD-fH_5hGDDqNvpRZk12_RhCN2WAe4_yaBamaNdtik/edit

Definitions of instrumented measures in the Shield experiment is here:

https://github.com/mozilla/testpilot-containers/blob/master/docs/metrics.md

Comment 1

[John Gruen [:jgruen]]

We are currently running the Containers addon in Test Pilot, and are collecting engagement and retention data (https://sql.telemetry.mozilla.org/dashboard/txp-containers-executive-summary).

In order to validate Containers as a privacy and tab management feature in Firefox we need to:
- run the add-on on the release population
- compare experiment retention and usage given two distinct onboarding flows
- collect data about Firefox usage in experiment and control cohorts in order to establish a control

The two experiment branches reflect two different onboarding flows: flow A is the flow that currently exists in the test pilot version of the add-on and describes privacy, and organizational benefits of Containers. flow B is more heavily focused on user security and blocking trackers. We would like to understand which flow results in higher feature retention.

The experiment data will be used to analyze measures documented here: 
https://github.com/mozilla/testpilot-containers/blob/master/docs/metrics.md

Additionally, we would like to understand from telemetry if certain types of users are more likely to engage with experiments. Do heavy tab users, heavy searchers, heavy add-on users, heavy hours users engage more with Containers proportionally more than lighter FF users. We'd also like to know if the presence of certain add-ons correlates more strongly with Containers retention. Having this information will help inform directed promotion of the feature in future.

Our hypothesis is that Containers will improve engagement measures in Firefox for some subset of privacy a/o task-management oriented users, we would like to identify signals form Telemetry that point to an abstractable user mode. We believe that having this data will help inform the shape of this product, and how it is implemented and marketed in Firefox.

Containers has ~5k dau in Test Pilot. I think we would like to see our two experimental cohorts equally split.

r? :rweiss

Comment 2

[John Gruen [:jgruen]]

Note:

Shield study of contianers should be run on release only. The only other filtering rule needs to be that users cannot be in the test pilot experiment. Add-on that would need be selected out of this experiment is = @testpilot-containers

Comment 3

[John Gruen [:jgruen]]

Actually, I suppose we should filter out all test pilot participants

Comment 4

[[e-mail]]

I'm a little concerned because I think the analysis that is presented here (as I understand it) is non-trivial.  It also feels a bit like you're jumping to more complicated analysis before you have the basics down.  I left some comments in the google doc linked above. 

I will tentatively r+ because your metrics.md is sufficiently documented according to our policies, and I think that most of my comments can be addressed with the data that you are proposing to collect alongside the existent telemetry environment data.  I will say that my r+ is also conditional on who is assigned to produce the analysis described in this docs.  :jgruen, who will be performing the analysis described in these docs?

Comment 5

[Jonathan Kingston [:jkt] he/him]

I'm glad we have called to question the outcomes here I know there has been a lot of skepticism of the results here in the team.

If we can improve the data captured to prove that users will be increased in satisfaction I would certainly rather delay this study to get genuine results.

Comment 6

[Matt Grimes [:Matt_G]]

This study has already shipped and ended. Resolving this bug.

Comment 7

[Justin Crawford [:hoosteeno] [:jcrawford]]

The containers experiment on Test Pilot eventually resulted in an extension, "Firefox Multi-Account Containers"[0]; i.e., the experiment code graduated to AMO. It has almost 500 reviews, mostly positive, many asking for additional features, and a 4+-star rating right now. The graduation report[1] contains some conclusions relevant to the study described by this bug.

In conversations with the experiment team, it sounds like the original hypotheses (linked in comment 0) were not proved or disproved during the experiment.  

[0] https://addons.mozilla.org/en-US/firefox/addon/multi-account-containers/
[1] https://medium.com/firefox-test-pilot/firefox-containers-are-go-ed2e3533b6e3