1478770 - [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62, 63, 64

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Chris More [:cmore]]

The Firefox Growth & Services team in conjunction with business, marketing, and engineering is conducting a VPN experiment in late Q3 for a sample of the US desktop population. This experiment is the first test of a number of core hypotheses of a potential future Premium Services product launch.

Comment 1

[Jens Hausdorf :jens1o [ni? me]]

wut.

I want Mozilla to be independent and not make any money of something.

Comment 2

[MarniePW [:marnie]]

[Tracking Requested - why for this release]: Will launch in release 62. Will likely roll into release 63.

Comment 3

[Chris More [:cmore]]

For more background information on this experiment, please check project links here: https://mana.mozilla.org/wiki/display/FIREFOX/Firefox+Growth+Team#FirefoxGrowthTeam-PremiumServices (look for the "2018 Q3/Q4 VPN Experiment" sub-heading)

Comment 4

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Study/rollout planned for 63, tracked.

Comment 5

[Chris More [:cmore]]

(In reply to Ritu Kothari (:ritu) from comment #4)
> Study/rollout planned for 63, tracked.

Hi Ritu. We are planning to launch at the end of Q3 or start of Q4, which is between the 62 and 62 window. We will likely be launching during Fx62.

Comment 6

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Thanks Chris for clarifying the timeline. This is tracked for both 62 and 63 so it should be on the radar for 62/63 release owners.

Comment 7

[Kamyar Ardekani (he/him)]

Attachment: GitHub PR for Firefox peer review

Comment 8

[MarniePW [:marnie]]

Aaron, when you're ready, can you add your r+ for the UX review of the doorhanger? Thanks!

Comment 9

[Aaron Benson]

Doorhanger design is looking great! Just a few notes.

Kamyar, can you take a look at: 

1) Vertically aligning the checkbox and the label ("Don't tell me ..") just a little more? The label looks like it is sitting just a pixel or two too low.

2) I think the doorhanger should go away if you click the application/hamburger menu.

3) Add a hover state for the (?) in the title bar. Sorry for not including this before, but it is in the spec now.

Thanks!

Comment 10

[Johann Hofmann [:johannh]]

As mentioned on Slack I can reasonably only leave comments on the full diff (including the first commit), to it would be great if you could update the review branch to be empty. Thanks!

Comment 11

[Aaron Benson]

I also noticed the doorhanger doesn't go away if you switch tabs or open up a new window. I think these are pretty strong indicators of a change in context and we should hide the doorhanger in those cases.

Comment 12

[Kamyar Ardekani (he/him)]

Attachment: GitHub PR for Firefox peer review

Comment 13

[Chris More [:cmore]]

There are going to be four treatment branches for this experiment and since we don't know how each branch will convert, we are going to do an even split among all four of them.

Comment 14

[Johann Hofmann [:johannh]]

Comment on attachment 9012669 [details] [diff] [review]
GitHub PR for Firefox peer review

Review of attachment 9012669 [details] [diff] [review]:
-----------------------------------------------------------------

Code is r=me

Please note that I sense a bit of brand risk here (specifically around showing what could be perceived as an advertisement in Firefox) and would like to flag this for sign-off from Firefox technical or product leadership. This is already mentioned in the PHD, I'd just like to reiterate.

Also note that I have not yet seen the support/info page for this, and I would consider a functional and easily understandable support page crucial in avoiding potential backlash.

Thank you!

Comment 15

[Darkspirit]

Hey, I would like to try this out before anything happens that could cause negative feedback because I am tired of Mozilla doing basically the right thing, but with avoidable problems. You do nice things and a VPN like Perfect Privacy but just from Mozilla would be great. Even this is just for the United States(?) it should be perceived positively elsewhere.

> Goal: End of Q3 2018, release 62, 63, 64

Key points without yet knowing what's going on or how to use the extension:
Firefox is not a whore, not "a product", where you can advertise things and do analytics like you could do for regular companies. Don't be greedy. Please go step by step. Just publishing a website and instantly advertising inside Firefox might be problematic. (Again, I don't know what you're planning.) Please publish the website at first, let interested users sign in to the VPN service via OpenID using their Firefox Accounts and make a cheap or free trial. Show us an entry on the account page that we're member of something. That subscribing also finances Firefox. ESNI should work as intended, so the VPN provider should validate DNSSEC and also support IPv6. Let us install the extension via AMO to get an impression. After you've got feedback from interested users you could start with en-US Nightly and gather further feedback before rolling it out to others.
Better go slow and maybe even only get sparse feedback from core users at first instead of instantly getting sceptical press.
Thank you.

Comment 16

[Pascal Chevrel:pascalc (PTO until April 26)]

(In reply to Chris More [:cmore] from comment #5)
> (In reply to Ritu Kothari (:ritu) from comment #4)
> > Study/rollout planned for 63, tracked.
> 
> Hi Ritu. We are planning to launch at the end of Q3 or start of Q4, which is
> between the 62 and 62 window. We will likely be launching during Fx62.

Hi Chris, could you update the schedule status for the experiment (we ship 63 within 2 weeks)? Should we keep on tracking it for 62? Thanks

Comment 17

[MarniePW [:marnie]]

Hi Pascal. The plan is to launch on Oct 22, the day before 63 ships. The experiment will run until January 22, 2019. So we'll need to track it for 62, 63, and 64.

Comment 18

[Chris More [:cmore]]

(In reply to Marnie Pasciuto-Wood [:marnie] from comment #17)
> Hi Pascal. The plan is to launch on Oct 22, the day before 63 ships. The
> experiment will run until January 22, 2019. So we'll need to track it for
> 62, 63, and 64.

Yes, what Marnie said. Thanks, Marnie! :)

Comment 19

[Kamyar Ardekani (he/him)]

Attachment: vpn_recommendation_shield_study_bug_1478770_-0.4.zip

version 0.4

Comment 20

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 21

[MarniePW [:marnie]]

Stephen, Devin, Chris, you need to get your data review started asap (for the doorhanger portion). Please answer the questions in this: https://github.com/mozilla/data-review/blob/master/request.md

Answer them in/by filing a bugzilla ticket. See full instructions here (look at section Step 1: Submit Request): https://wiki.mozilla.org/Firefox/Data_Collection

NI Rob Rayborn (rrayborn) in the bugzilla bug. We'll need to get this started very soon to hit the Oct 22 date.

Comment 22

[Chris More [:cmore]]

(In reply to Marnie Pasciuto-Wood [:marnie] from comment #21)
> Stephen, Devin, Chris, you need to get your data review started asap (for
> the doorhanger portion). Please answer the questions in this:
> https://github.com/mozilla/data-review/blob/master/request.md
> 
> Answer them in/by filing a bugzilla ticket. See full instructions here (look
> at section Step 1: Submit Request):
> https://wiki.mozilla.org/Firefox/Data_Collection
> 
> NI Rob Rayborn (rrayborn) in the bugzilla bug. We'll need to get this
> started very soon to hit the Oct 22 date.

Done! Bug 1499081 filed.

Comment 23

[Chris More [:cmore]]

Osunick/Reese: I need your +1 to ship this experiment on Oct 22nd pending on all of the final QA and release drivers sign off. 

PHD: https://docs.google.com/document/d/14Rq1sfKYg8U2_-UKd3KR6qLBuBzsoEt6oSze4G5o4Ns/edit?ts=5b58bad9

Data review: https://docs.google.com/document/d/1iHp9-11wDw8hsa7-5FwWLgd2ewiu10EqRZCcsIa4zOc/edit#

We are still wrapping up a number of improvements to the experiments, but we need to wrap up this internal due diligence. A simple +1 or r+ is suffice as a bug comment. 

Thanks in advance!

Comment 24

[Marissa (Reese) Wood]

+1 :)

Comment 25

[Aaron Benson]

Doorhanger UX looks great! r+

Comment 26

[Nick Nguyen [:osunick]]

Approved.

Comment 27

[Ilana]

Science: R+

Comment 28

[Kamyar Ardekani (he/him)]

Attachment: vpn_recommendation_shield_study_bug_1478770_-0.6.zip

Comment 29

[Pascal Chevrel:pascalc (PTO until April 26)]

This is planned to be deployed at the same time as the Block Autoplay Shield study (bug 1475099) which also displays a door-hanger for sites streaming audio and video. Should we make sure the two studies don't enroll the same users to avoid having 2 door-hangers each proposing an action on the same streaming site?

Comment 30

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 31

[Paul Theriault [:pauljt] (no longer reading bugmail)]

FYI all that security review of the Shield Extension is completed in bug 1486379. The review was of 0.5 (a couple days ago) but I understand there has not been any significant changes since then.

Comment 32

[MarniePW [:marnie]]

Pascal, good question. We can build exclusions into the recipe so that the experiments with doorhangers don't get deployed to the same users.

Also, we are aiming to launch this experiment on Monday, October 22nd. On the very slim chance that something comes up to preclude us from launching on Monday, I'd like to plan ahead for a slip to launching Wednesday, October 24. Do you (or Ritu) have any concerns about the possible date change from a release management perspective? We wouldn't launch on the 23rd due to the 63 launch.

Comment 33

[Michael Feldman (use NI)]

legal is signed off on the doorhanger UX

Comment 34

[Pascal Chevrel:pascalc (PTO until April 26)]

+1 from Relman and no problem for a launch on October 24 instead of 22.

Comment 35

[Kamyar Ardekani (he/him)]

Attachment: firefox_vpn_study-0.7.zip

Comment 36

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 37

[Kamyar Ardekani (he/him)]

Attachment: firefox_vpn_study-0.8.zip

Comment 38

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 39

[Kamyar Ardekani (he/him)]

Attachment: firefox_vpn_study-0.9.zip

Comment 40

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 41

[Ciprian Muresan [:cmuresan], Ecosystem QA]

We have finalized testing the Premium Services VPN experiment Shield Study experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:
- All issues that were found have been fixed and verified.   

However, the redirect link is still the one for the Dev version of the website and not the one for the Prod version. This change is currently tracked in #72 (https://github.com/raymak/vpn-recommendation-shield-study/pull/72)

Testing Summary:
- Test suite: Windows 10(https://testrail.stage.mozaws.net/index.php?/runs/view/13099) & MacOS 10.13 (https://testrail.stage.mozaws.net/index.php?/runs/view/13100)

Tested Platforms:
- Windows 10 x64
- Mac 10.13.3

Tested Firefox versions:
- Firefox Release build v62.0.3;

Comment 42

[Kamyar Ardekani (he/him)]

Attachment: firefox_vpn_study-1.0.zip

This version (1.0) is functionally equivalent to version 0.9 that has been approved by QA. The minimal changes are only to prepare the addon for the final deployment. Changes are limited to:

- Final landing page has been set to Prod: https://premium.firefox.com/vpn/
- Telemetry testing flag has been set to "false"
- The version of the addon has been set to 1.0

Comment 43

[Kamyar Ardekani (he/him)]

Devin, would you mind downloading the v1.0 and v0.9 zip files and comparing them using diff to ensure that the changes are only those that set the release parameters? I would like this to be on the record.

Comment 44

[Devin Reams]

Hey Kamyar, I confirmed the 0.9 and 1.0 versions are the same except for the version, landing URL, and testing flag changes.

~/Downloads $ diff -r firefox_vpn_study-0.9 firefox_vpn_study-1.0
diff -r firefox_vpn_study-0.9/manifest.json firefox_vpn_study-1.0/manifest.json
4c4
<   "version": "0.9",
---
>   "version": "1.0",
diff -r firefox_vpn_study-0.9/privileged/vpnRecommender/api.js firefox_vpn_study-1.0/privileged/vpnRecommender/api.js
52c52
< const VPN_LANDING_PAGE_URL = "https://dev.psvpn.nonprod.cloudops.mozgcp.net/";
---
> const VPN_LANDING_PAGE_URL = "https://premium.firefox.com/vpn/";
diff -r firefox_vpn_study-0.9/studySetup.js firefox_vpn_study-1.0/studySetup.js
38c38
<     removeTestingFlag: false,
---
>     removeTestingFlag: true,

Comment 45

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 46

[Robert (rrayborn, he/him)]

We're live at 0.1% of the target population.  Gradual ramp up to 0.5% expected if things look good.

Comment 47

[Chris More [:cmore]]

Rob: Request to move from 0.5% sample to 1% sample.

We got confirmation from Jesse McCrosky on the data science team that no measurable negative impact to retention can be seen.

Thanks!

Comment 48

[Robert (rrayborn, he/him)]

We're live with the bump.

Comment 49

[Josh Gaunt [:jgaunt]]

Recipe 619 has been disabled, per cmore request - this study has concluded.