[Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62, 63, 64

NEW
Assigned to

Status

8 months ago
2 months ago

People

(Reporter: cmore, Assigned: kardekani, NeedInfo)

Tracking

(Depends on: 1 bug)

unspecified
Dependency tree / graph
Bug Flags:
shield-qa +

Firefox Tracking Flags

(firefox61 unaffected, firefox62+ fixed, firefox63+ fixed, firefox64+ fixed)

Details

(Whiteboard: Github repository of the study addon, URL)

Attachments

(2 attachments, 12 obsolete attachments)

378.25 KB, application/zip
Details
383.37 KB, application/x-xpinstall
Details
(Reporter)

Description

8 months ago
The Firefox Growth & Services team in conjunction with business, marketing, and engineering is conducting a VPN experiment in late Q3 for a sample of the US desktop population. This experiment is the first test of a number of core hypotheses of a potential future Premium Services product launch.
(Reporter)

Updated

8 months ago
Comment hidden (offtopic)
status-firefox61: --- → unaffected
status-firefox62: --- → affected
status-firefox63: --- → affected
(Reporter)

Updated

8 months ago
Depends on: 1480156
[Tracking Requested - why for this release]: Will launch in release 62. Will likely roll into release 63.
tracking-firefox62: --- → ?
tracking-firefox63: --- → ?
Summary: [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018 → [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62
Summary: [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62 → [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62 and 63
(Reporter)

Updated

7 months ago
Depends on: 1484079
tracking-firefox62: ? → +
(Reporter)

Comment 3

7 months ago
For more background information on this experiment, please check project links here: https://mana.mozilla.org/wiki/display/FIREFOX/Firefox+Growth+Team#FirefoxGrowthTeam-PremiumServices (look for the "2018 Q3/Q4 VPN Experiment" sub-heading)
Depends on: 1484576
Depends on: 1484847

Updated

7 months ago
Depends on: 1485058
Study/rollout planned for 63, tracked.
tracking-firefox63: ? → +
(Reporter)

Comment 5

7 months ago
(In reply to Ritu Kothari (:ritu) from comment #4)
> Study/rollout planned for 63, tracked.

Hi Ritu. We are planning to launch at the end of Q3 or start of Q4, which is between the 62 and 62 window. We will likely be launching during Fx62.
Flags: needinfo?(rkothari)
Thanks Chris for clarifying the timeline. This is tracked for both 62 and 63 so it should be on the radar for 62/63 release owners.
Flags: needinfo?(rkothari)
Depends on: 1486379
(Assignee)

Comment 7

6 months ago
Posted file GitHub PR for Firefox peer review (obsolete) —
Assignee: nobody → kardekani
Flags: needinfo?(jhofmann)
Aaron, when you're ready, can you add your r+ for the UX review of the doorhanger? Thanks!
Flags: needinfo?(abenson)
status-firefox64: --- → affected
tracking-firefox64: --- → ?
Summary: [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62 and 63 → [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62, 63, 64
tracking-firefox64: ? → +

Comment 9

6 months ago
Doorhanger design is looking great! Just a few notes.

Kamyar, can you take a look at: 

1) Vertically aligning the checkbox and the label ("Don't tell me ..") just a little more? The label looks like it is sitting just a pixel or two too low.

2) I think the doorhanger should go away if you click the application/hamburger menu.

3) Add a hover state for the (?) in the title bar. Sorry for not including this before, but it is in the spec now.

Thanks!

Updated

6 months ago
Flags: needinfo?(abenson) → needinfo?(kamyar.ardekani)
As mentioned on Slack I can reasonably only leave comments on the full diff (including the first commit), to it would be great if you could update the review branch to be empty. Thanks!
Flags: needinfo?(jhofmann)

Comment 11

6 months ago
I also noticed the doorhanger doesn't go away if you switch tabs or open up a new window. I think these are pretty strong indicators of a change in context and we should hide the doorhanger in those cases.
(Assignee)

Comment 12

6 months ago
Attachment #9012167 - Attachment is obsolete: true
Flags: needinfo?(jhofmann)
Flags: needinfo?(jhofmann)
Attachment #9012669 - Flags: review?(jhofmann)

Updated

6 months ago
Depends on: 1495090

Updated

6 months ago
Depends on: 1495092

Updated

6 months ago
Depends on: 1495105
(Reporter)

Comment 13

6 months ago
There are going to be four treatment branches for this experiment and since we don't know how each branch will convert, we are going to do an even split among all four of them.
Comment on attachment 9012669 [details] [diff] [review]
GitHub PR for Firefox peer review

Review of attachment 9012669 [details] [diff] [review]:
-----------------------------------------------------------------

Code is r=me

Please note that I sense a bit of brand risk here (specifically around showing what could be perceived as an advertisement in Firefox) and would like to flag this for sign-off from Firefox technical or product leadership. This is already mentioned in the PHD, I'd just like to reiterate.

Also note that I have not yet seen the support/info page for this, and I would consider a functional and easily understandable support page crucial in avoiding potential backlash.

Thank you!
Attachment #9012669 - Attachment is patch: true
Attachment #9012669 - Attachment mime type: text/x-github-pull-request → text/plain
Attachment #9012669 - Flags: review?(jhofmann) → review+
Hey, I would like to try this out before anything happens that could cause negative feedback because I am tired of Mozilla doing basically the right thing, but with avoidable problems. You do nice things and a VPN like Perfect Privacy but just from Mozilla would be great. Even this is just for the United States(?) it should be perceived positively elsewhere.

> Goal: End of Q3 2018, release 62, 63, 64

Key points without yet knowing what's going on or how to use the extension:
Firefox is not a whore, not "a product", where you can advertise things and do analytics like you could do for regular companies. Don't be greedy. Please go step by step. Just publishing a website and instantly advertising inside Firefox might be problematic. (Again, I don't know what you're planning.) Please publish the website at first, let interested users sign in to the VPN service via OpenID using their Firefox Accounts and make a cheap or free trial. Show us an entry on the account page that we're member of something. That subscribing also finances Firefox. ESNI should work as intended, so the VPN provider should validate DNSSEC and also support IPv6. Let us install the extension via AMO to get an impression. After you've got feedback from interested users you could start with en-US Nightly and gather further feedback before rolling it out to others.
Better go slow and maybe even only get sparse feedback from core users at first instead of instantly getting sceptical press.
Thank you.
(In reply to Chris More [:cmore] from comment #5)
> (In reply to Ritu Kothari (:ritu) from comment #4)
> > Study/rollout planned for 63, tracked.
> 
> Hi Ritu. We are planning to launch at the end of Q3 or start of Q4, which is
> between the 62 and 62 window. We will likely be launching during Fx62.

Hi Chris, could you update the schedule status for the experiment (we ship 63 within 2 weeks)? Should we keep on tracking it for 62? Thanks
Flags: needinfo?(chrismore.bugzilla)
Hi Pascal. The plan is to launch on Oct 22, the day before 63 ships. The experiment will run until January 22, 2019. So we'll need to track it for 62, 63, and 64.
Flags: needinfo?(chrismore.bugzilla)
(Reporter)

Comment 18

6 months ago
(In reply to Marnie Pasciuto-Wood [:marnie] from comment #17)
> Hi Pascal. The plan is to launch on Oct 22, the day before 63 ships. The
> experiment will run until January 22, 2019. So we'll need to track it for
> 62, 63, and 64.

Yes, what Marnie said. Thanks, Marnie! :)
(Assignee)

Comment 19

6 months ago
version 0.4
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
No longer depends on: 1486379
Stephen, Devin, Chris, you need to get your data review started asap (for the doorhanger portion). Please answer the questions in this: https://github.com/mozilla/data-review/blob/master/request.md

Answer them in/by filing a bugzilla ticket. See full instructions here (look at section Step 1: Submit Request): https://wiki.mozilla.org/Firefox/Data_Collection

NI Rob Rayborn (rrayborn) in the bugzilla bug. We'll need to get this started very soon to hit the Oct 22 date.
Flags: needinfo?(slangtonhood)
Flags: needinfo?(dreams)
Flags: needinfo?(chrismore.bugzilla)
(Reporter)

Comment 22

5 months ago
(In reply to Marnie Pasciuto-Wood [:marnie] from comment #21)
> Stephen, Devin, Chris, you need to get your data review started asap (for
> the doorhanger portion). Please answer the questions in this:
> https://github.com/mozilla/data-review/blob/master/request.md
> 
> Answer them in/by filing a bugzilla ticket. See full instructions here (look
> at section Step 1: Submit Request):
> https://wiki.mozilla.org/Firefox/Data_Collection
> 
> NI Rob Rayborn (rrayborn) in the bugzilla bug. We'll need to get this
> started very soon to hit the Oct 22 date.

Done! Bug 1499081 filed.
Flags: needinfo?(slangtonhood)
Flags: needinfo?(dreams)
Flags: needinfo?(chrismore.bugzilla)
(Reporter)

Updated

5 months ago
Depends on: 1499081
(Reporter)

Comment 23

5 months ago
Osunick/Reese: I need your +1 to ship this experiment on Oct 22nd pending on all of the final QA and release drivers sign off. 

PHD: https://docs.google.com/document/d/14Rq1sfKYg8U2_-UKd3KR6qLBuBzsoEt6oSze4G5o4Ns/edit?ts=5b58bad9

Data review: https://docs.google.com/document/d/1iHp9-11wDw8hsa7-5FwWLgd2ewiu10EqRZCcsIa4zOc/edit#

We are still wrapping up a number of improvements to the experiments, but we need to wrap up this internal due diligence. A simple +1 or r+ is suffice as a bug comment. 

Thanks in advance!
Flags: needinfo?(nnguyen)
Flags: needinfo?(mmorris)

Comment 24

5 months ago
+1 :)
Flags: needinfo?(mmorris)

Comment 25

5 months ago
Doorhanger UX looks great! r+
Approved.
Flags: needinfo?(nnguyen)

Comment 27

5 months ago
Science: R+
(Assignee)

Comment 28

5 months ago
Attachment #9015395 - Attachment is obsolete: true
Attachment #9015396 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
This is planned to be deployed at the same time as the Block Autoplay Shield study (bug 1475099) which also displays a door-hanger for sites streaming audio and video. Should we make sure the two studies don't enroll the same users to avoid having 2 door-hangers each proposing an action on the same streaming site?
Flags: needinfo?(mcooper)
FYI all that security review of the Shield Extension is completed in bug 1486379. The review was of 0.5 (a couple days ago) but I understand there has not been any significant changes since then.
Pascal, good question. We can build exclusions into the recipe so that the experiments with doorhangers don't get deployed to the same users.

Also, we are aiming to launch this experiment on Monday, October 22nd. On the very slim chance that something comes up to preclude us from launching on Monday, I'd like to plan ahead for a slip to launching Wednesday, October 24. Do you (or Ritu) have any concerns about the possible date change from a release management perspective? We wouldn't launch on the 23rd due to the 63 launch.
Flags: needinfo?(rkothari)
Flags: needinfo?(pascalc)
legal is signed off on the doorhanger UX
+1 from Relman and no problem for a launch on October 24 instead of 22.
Flags: needinfo?(pascalc)

Updated

5 months ago
Flags: needinfo?(rkothari)
(Assignee)

Comment 35

5 months ago
Posted file firefox_vpn_study-0.7.zip (obsolete) —
Attachment #9012669 - Attachment is obsolete: true
Attachment #9017735 - Attachment is obsolete: true
Attachment #9017943 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
(Assignee)

Comment 37

5 months ago
Posted file firefox_vpn_study-0.8.zip (obsolete) —
Attachment #9018453 - Attachment is obsolete: true
Attachment #9018456 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
(Assignee)

Comment 39

5 months ago
Posted file firefox_vpn_study-0.9.zip (obsolete) —
Attachment #9019120 - Attachment is obsolete: true
Attachment #9019129 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
Flags: needinfo?(mcooper)
We have finalized testing the Premium Services VPN experiment Shield Study experiment.

QA’s recommendation: GREEN - SHIP IT

Reasoning:
- All issues that were found have been fixed and verified.   

However, the redirect link is still the one for the Dev version of the website and not the one for the Prod version. This change is currently tracked in #72 (https://github.com/raymak/vpn-recommendation-shield-study/pull/72)

Testing Summary:
- Test suite: Windows 10(https://testrail.stage.mozaws.net/index.php?/runs/view/13099) & MacOS 10.13 (https://testrail.stage.mozaws.net/index.php?/runs/view/13100)

Tested Platforms:
- Windows 10 x64
- Mac 10.13.3

Tested Firefox versions:
- Firefox Release build v62.0.3;
Flags: shield-qa+
(Assignee)

Comment 42

5 months ago
This version (1.0) is functionally equivalent to version 0.9 that has been approved by QA. The minimal changes are only to prepare the addon for the final deployment. Changes are limited to:

- Final landing page has been set to Prod: https://premium.firefox.com/vpn/
- Telemetry testing flag has been set to "false"
- The version of the addon has been set to 1.0
Attachment #9019195 - Attachment is obsolete: true
Attachment #9019199 - Attachment is obsolete: true
Flags: needinfo?(mcooper)
(Assignee)

Comment 43

5 months ago
Devin, would you mind downloading the v1.0 and v0.9 zip files and comparing them using diff to ensure that the changes are only those that set the release parameters? I would like this to be on the record.
Flags: needinfo?(dreams)
Hey Kamyar, I confirmed the 0.9 and 1.0 versions are the same except for the version, landing URL, and testing flag changes.

~/Downloads $ diff -r firefox_vpn_study-0.9 firefox_vpn_study-1.0
diff -r firefox_vpn_study-0.9/manifest.json firefox_vpn_study-1.0/manifest.json
4c4
<   "version": "0.9",
---
>   "version": "1.0",
diff -r firefox_vpn_study-0.9/privileged/vpnRecommender/api.js firefox_vpn_study-1.0/privileged/vpnRecommender/api.js
52c52
< const VPN_LANDING_PAGE_URL = "https://dev.psvpn.nonprod.cloudops.mozgcp.net/";
---
> const VPN_LANDING_PAGE_URL = "https://premium.firefox.com/vpn/";
diff -r firefox_vpn_study-0.9/studySetup.js firefox_vpn_study-1.0/studySetup.js
38c38
<     removeTestingFlag: false,
---
>     removeTestingFlag: true,
Flags: needinfo?(dreams)
Flags: needinfo?(mcooper)
We're live at 0.1% of the target population.  Gradual ramp up to 0.5% expected if things look good.
(Assignee)

Updated

5 months ago
Whiteboard: Github repository of the study addon
(Reporter)

Updated

5 months ago
Depends on: 1502456
(Reporter)

Comment 47

5 months ago
Rob: Request to move from 0.5% sample to 1% sample.

We got confirmation from Jesse McCrosky on the data science team that no measurable negative impact to retention can be seen.

Thanks!
Flags: needinfo?(rrayborn)
We're live with the bump.
Flags: needinfo?(rrayborn)

Recipe 619 has been disabled, per cmore request - this study has concluded.

status-firefox62: affected → fixed
status-firefox63: affected → fixed
status-firefox64: affected → fixed
You need to log in before you can comment on or make changes to this bug.