1420373 - [Shield] Opt-out Study: Onboarding Notification A/B testing

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[chsiang]

Basic description of experiment: 
Based on a qualitative user research, notifications are not effective in engaging users to onboarding tour. We would like to use shield study to test this to a larger group of users.

What is the preference we will be changing? 
Browser.onboarding.notification.mute-duration-on-first-session-ms
browser.onboarding.notification.tour-ids-queue    

What are the branches of the study and what values should each branch be set to? 
Control: 
No change
Treatment 
Browser.onboarding.notification.mute-duration-on-first-session-ms set to 1209600000 (default is set to 300000)
clearUserPerf browser.onboarding.notification.tour-ids-queue

What percentage of users do you want in each branch? 
Need input from Product Data Science or Strategy and Insights. Two groups should be equal size. 


What Channels and locales do you intend to ship to? 
All channels and locales 
Release 57

What is your intended go live date and how long will the study run? 
12/4-12/15

Are there specific criteria for participants? 
No

What is the main effect you are looking for and what data will you use to make these decisions? 
When the treatment group is equal or greater than the control group in 
Total overlay opened
Total overlay CTA clicks

Who is the owner of the data analysis for this study? 
Need feedback 

Will this experiment require uplift? 
No

QA Status of your code: Green, yellow, red. Your code should be QA’d to ensure that changing the preference values has the intended effect you are looking for and does not cause obvious regressions to Firefox. All experiments must pass QA. Depending on the channel/population size a dev QA may be accepted.

Do you plan on surveying users at the end of the study? 
Yes

For Control
-Have you seen the Firefox logo? (logo image)
-What do you think it is?
-Do you find it (5 point scale)
  -Undiscoverable to Noticeable
  -Confusing to Understandable
  -Not helpful to Helpful

-Have you seen the Notification? (notification image)
-What do you think it is?
-Do you find it (5 point scale)
  -Undiscoverable to Noticeable
  -Confusing to Understandable
  -Not helpful to Helpful

For Treatment
-Have you seen the Firefox logo? (logo image)
-What do you think it is?
-Do you find it (5 point scale)
  -Undiscoverable to Noticeable
  -Confusing to Understandable
  -Not helpful to Helpful

Comment 1

[[e-mail]]

@glind, @sguha we'd like to launch a onboarding study, could you help check on this?

Comment 2

[[e-mail]]

Attachment: [e-mail]

Create a shield-study compatible add-on for Onboarding Notification Experiment so we can do Shield study by it.

The addon source is on https://github.com/mozilla/shield-onboarding thanks to Gregg's example. It can be run under firefox 57 from about:debugging, and can choose variations based on Cindy's proposal.

Will send PI-request (for Product Integrity) before signing https://mana.mozilla.org/wiki/display/PI/PI+Request

Comment 3

[Rehan Dalal (Inactive)]

mythmon, could you sign this?

Comment 4

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 5

[[e-mail]]

Attachment: linked-addon.xpi

append linked-addon.xpi for debugging

Comment 6

[[e-mail]]

Attachment: [e-mail]

Updated the addon to fill the bug number in package.json and fixed a potential issue. 

Cristian Comorasu will be responsible for QA testing.

Will send a PR for review as well before signing.

Comment 7

[[e-mail]]

Attachment: shield-onboarding-addon.patch

Though it seems not essential, I'd like to get some review since the addon template is changing recently.

Comment 8

[[e-mail]]

FYR the addon source is located in https://github.com/mozilla/shield-onboarding/commits/master

Comment 9

[[e-mail]]

Cristian and I are cooperate for QA testing plan at etherpad https://public.etherpad-mozilla.org/p/onboarding_shield_QA

Feel free to comment or add suggestions

Comment 10

[[e-mail]]

(In reply to Fred Lin [:gasolin] from comment #6)
> Created attachment 8932313 [details]
> [e-mail]
> 
> Updated the addon to fill the bug number in package.json and fixed a
> potential issue. 
> 
> Cristian Comorasu will be responsible for QA testing.
> 
> Will send a PR for review as well before signing.

mythmon, could you help sign this (1.3.0)?


From Christian "(while test 1.3.0.xpi) There was an error during installation: Add-on [e-mail] is not compatible with application version. add-on minVersion: 57.0. add-on maxVersion: *". on Fx 57.0-build4, however it works fine on Fx 57.0b14 dev edition"

Comment 11

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 12

[Matt Grimes [:Matt_G]]

Can we get a recap of the results and then have you close this bug?

Comment 13

[chsiang]

Hi Matt,
If I recall correctly, this test was pushed to the backlog so the shield team could focus on higher priority projects. 
Cindy