[Shield] Opt-in Study: Pioneer Cohort



a year ago
7 months ago


(Reporter: telliott, Assigned: telliott)


Firefox Tracking Flags

(Not tracked)



(1 attachment)



a year ago
+++ This bug was initially created as a clone of Bug #1379251 +++

Basic description of experiment: This is the empty addon to use as a marker for people who have opted to join the Pioneer cohort

What are the branches of the study? There are no branches. This will be made available to all as an opt-in

What Channels/locales do you intend to ship to? EN locales.

What is your intended go live date and how long will the study run? Exact date TBD, but probably next week. Setting this up so it can launch in my absence. No earlier than 7/25.

Are there specific criteria for participants? No

What is the main effect you are looking for and what data will you use to make these decisions? Some users will choose to install the addon. We'll be able to target them later with Pioneer.

Who is the owner of the data analysis for this study? There is no data.

Who will have access to the data? There is no data

Do you plan on surveying users at the end of the study? No

QA Status of your add-on: Unreviewed, but it's empty, which makes me think it's probably green already :)

Link to any relevant google docs / Drive files that describe the project. Links to prior art if it exists: https://docs.google.com/document/d/19M0smiBbnE_G0J0Txix2SfXXSMD3GbFngQICr4fDK4g/edit


a year ago
Flags: needinfo?(mgrimes)
Looks good to me. Do you have consent page language? Since we are still using AMO temporarily for hosting we'll need to check that page in by Monday and then we could go live thurs.
Flags: needinfo?(mgrimes)
Depends on: 1383098
Approved per Release Drivers.
Created attachment 8910082 [details]

We're going to update this study with an improved opt-in experience. We'll be using the opt-out mechanism to ship it, but it is still opt-in enrollment to the Pioneer program.

The study repo is https://github.com/mozilla/pioneer-enrollment-study and the README there contains implementation and data collection details. Since we are only collecting interaction data with the prompts, and since this is an update to an existing study, we do not need a data review or re-approval from S&I. I've confirmed with Matt Grimes that we can release this new variation.

Our initial run will be 10% of Beta users (UI differences will require a different add-on for 57). I sign-off as having tested the study on MacOS, Windows, and Linux in Beta 56 (S&I also helped test functionality). The study itself takes up to 5 days to run after first enrollment; we'll be running until Beta switches over to 57, which AFAICT happens on the 27th, so we should get 3-4 days of enrollment.

The attached add-on's manifest limits it to running on 56 only.
I have signed and uploaded the XPI from comment 4 to Normandy, and it is available for use in recipes.
The recipe has been enabled for Beta 56 at a 10% sample of users who haven't already enrolled in Pioneer.
New enrollment has been paused, since the experiment runs for 4-5 days, and next Tuesday we'll be upgrading users to release.
The recipe has been disabled. I'll be working with jgaunt to analyze the data.
Depends on: 1404307
Depends on: 1404312
Depends on: 1404319
Depends on: 1404332
Depends on: 1404333
New signed build of the add-on is up at https://net-mozaws-prod-addons-signxpi-output.s3.amazonaws.com/pioneer-enrollment-study-v2.0.2.xpi with fixes for bug 1404332, bug 1404319, and bug 1404333.
I've added a version of the build from comment 10 to Normandy's production instance with a testing filter. You can trigger Shield to download and install this study with the following steps:

1. Create a new profile and start up Firefox.
2. Open about:config. Set the extensions.shield-recipe-client.dev_mode pref to true, and the extensions.shield-recipe-client.user_id pref to the string "pioneer-quick-timer-2".
3. Restart Firefox.

After restarting, you should see the first prompt from the study appear very soon after first run. Note that you will probably also hit the symptoms from bug 1404312 if you follow the steps above.
Depends on: 1404908
No longer depends on: 1404908
I have uploaded version 2.0.2 of the Pioneer Enrollment Study to Normandy under the name "Pioneer Enrollment Study v2.0.2". The XPI I uploaded was taken from https://net-mozaws-prod-addons-signxpi-output.s3.amazonaws.com/pioneer-enrollment-study-v2.0.2.xpi.
QE verified functionality of the add-on via email and consider it complete (taking into consideration caveats in the still-open bug 1404307 and bug 1404312). Test results are available: https://public.etherpad-mozilla.org/p/PI-Request_PioneerEnrollmentStudy_v2.0.0
The recipe has been updated to target 1% of release (en-* locales, limited to 56 only) and has been published.
No longer depends on: 1404312


7 months ago
Last Resolved: 7 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.