The Firefox Growth & Services team in conjunction with business, marketing, and engineering is conducting a VPN experiment in late Q3 for a sample of the US desktop population. This experiment is the first test of a number of core hypotheses of a potential future Premium Services product launch.
status-firefox61: --- → unaffected
status-firefox62: --- → affected
status-firefox63: --- → affected
[Tracking Requested - why for this release]: Will launch in release 62. Will likely roll into release 63.
tracking-firefox62: --- → ?
tracking-firefox63: --- → ?
Summary: [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018 → [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62
Summary: [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62 → [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62 and 63
For more background information on this experiment, please check project links here: https://mana.mozilla.org/wiki/display/FIREFOX/Firefox+Growth+Team#FirefoxGrowthTeam-PremiumServices (look for the "2018 Q3/Q4 VPN Experiment" sub-heading)
Study/rollout planned for 63, tracked.
tracking-firefox63: ? → +
(In reply to Ritu Kothari (:ritu) from comment #4) > Study/rollout planned for 63, tracked. Hi Ritu. We are planning to launch at the end of Q3 or start of Q4, which is between the 62 and 62 window. We will likely be launching during Fx62.
Thanks Chris for clarifying the timeline. This is tracked for both 62 and 63 so it should be on the radar for 62/63 release owners.
Assignee: nobody → kardekani
Aaron, when you're ready, can you add your r+ for the UX review of the doorhanger? Thanks!
status-firefox64: --- → affected
tracking-firefox64: --- → ?
Summary: [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62 and 63 → [Shield] Conduct first Premium Services VPN Experiment -- Goal: End of Q3 2018, release 62, 63, 64
Doorhanger design is looking great! Just a few notes. Kamyar, can you take a look at: 1) Vertically aligning the checkbox and the label ("Don't tell me ..") just a little more? The label looks like it is sitting just a pixel or two too low. 2) I think the doorhanger should go away if you click the application/hamburger menu. 3) Add a hover state for the (?) in the title bar. Sorry for not including this before, but it is in the spec now. Thanks!
As mentioned on Slack I can reasonably only leave comments on the full diff (including the first commit), to it would be great if you could update the review branch to be empty. Thanks!
I also noticed the doorhanger doesn't go away if you switch tabs or open up a new window. I think these are pretty strong indicators of a change in context and we should hide the doorhanger in those cases.
Attachment #9012167 - Attachment is obsolete: true
Attachment #9012669 - Flags: review?(jhofmann)
There are going to be four treatment branches for this experiment and since we don't know how each branch will convert, we are going to do an even split among all four of them.
Comment on attachment 9012669 [details] [diff] [review] GitHub PR for Firefox peer review Review of attachment 9012669 [details] [diff] [review]: ----------------------------------------------------------------- Code is r=me Please note that I sense a bit of brand risk here (specifically around showing what could be perceived as an advertisement in Firefox) and would like to flag this for sign-off from Firefox technical or product leadership. This is already mentioned in the PHD, I'd just like to reiterate. Also note that I have not yet seen the support/info page for this, and I would consider a functional and easily understandable support page crucial in avoiding potential backlash. Thank you!
Hey, I would like to try this out before anything happens that could cause negative feedback because I am tired of Mozilla doing basically the right thing, but with avoidable problems. You do nice things and a VPN like Perfect Privacy but just from Mozilla would be great. Even this is just for the United States(?) it should be perceived positively elsewhere. > Goal: End of Q3 2018, release 62, 63, 64 Key points without yet knowing what's going on or how to use the extension: Firefox is not a whore, not "a product", where you can advertise things and do analytics like you could do for regular companies. Don't be greedy. Please go step by step. Just publishing a website and instantly advertising inside Firefox might be problematic. (Again, I don't know what you're planning.) Please publish the website at first, let interested users sign in to the VPN service via OpenID using their Firefox Accounts and make a cheap or free trial. Show us an entry on the account page that we're member of something. That subscribing also finances Firefox. ESNI should work as intended, so the VPN provider should validate DNSSEC and also support IPv6. Let us install the extension via AMO to get an impression. After you've got feedback from interested users you could start with en-US Nightly and gather further feedback before rolling it out to others. Better go slow and maybe even only get sparse feedback from core users at first instead of instantly getting sceptical press. Thank you.
(In reply to Chris More [:cmore] from comment #5) > (In reply to Ritu Kothari (:ritu) from comment #4) > > Study/rollout planned for 63, tracked. > > Hi Ritu. We are planning to launch at the end of Q3 or start of Q4, which is > between the 62 and 62 window. We will likely be launching during Fx62. Hi Chris, could you update the schedule status for the experiment (we ship 63 within 2 weeks)? Should we keep on tracking it for 62? Thanks
Hi Pascal. The plan is to launch on Oct 22, the day before 63 ships. The experiment will run until January 22, 2019. So we'll need to track it for 62, 63, and 64.
(In reply to Marnie Pasciuto-Wood [:marnie] from comment #17) > Hi Pascal. The plan is to launch on Oct 22, the day before 63 ships. The > experiment will run until January 22, 2019. So we'll need to track it for > 62, 63, and 64. Yes, what Marnie said. Thanks, Marnie! :)
Stephen, Devin, Chris, you need to get your data review started asap (for the doorhanger portion). Please answer the questions in this: https://github.com/mozilla/data-review/blob/master/request.md Answer them in/by filing a bugzilla ticket. See full instructions here (look at section Step 1: Submit Request): https://wiki.mozilla.org/Firefox/Data_Collection NI Rob Rayborn (rrayborn) in the bugzilla bug. We'll need to get this started very soon to hit the Oct 22 date.
(In reply to Marnie Pasciuto-Wood [:marnie] from comment #21) > Stephen, Devin, Chris, you need to get your data review started asap (for > the doorhanger portion). Please answer the questions in this: > https://github.com/mozilla/data-review/blob/master/request.md > > Answer them in/by filing a bugzilla ticket. See full instructions here (look > at section Step 1: Submit Request): > https://wiki.mozilla.org/Firefox/Data_Collection > > NI Rob Rayborn (rrayborn) in the bugzilla bug. We'll need to get this > started very soon to hit the Oct 22 date. Done! Bug 1499081 filed.
Osunick/Reese: I need your +1 to ship this experiment on Oct 22nd pending on all of the final QA and release drivers sign off. PHD: https://docs.google.com/document/d/14Rq1sfKYg8U2_-UKd3KR6qLBuBzsoEt6oSze4G5o4Ns/edit?ts=5b58bad9 Data review: https://docs.google.com/document/d/1iHp9-11wDw8hsa7-5FwWLgd2ewiu10EqRZCcsIa4zOc/edit# We are still wrapping up a number of improvements to the experiments, but we need to wrap up this internal due diligence. A simple +1 or r+ is suffice as a bug comment. Thanks in advance!
Doorhanger UX looks great! r+
This is planned to be deployed at the same time as the Block Autoplay Shield study (bug 1475099) which also displays a door-hanger for sites streaming audio and video. Should we make sure the two studies don't enroll the same users to avoid having 2 door-hangers each proposing an action on the same streaming site?
FYI all that security review of the Shield Extension is completed in bug 1486379. The review was of 0.5 (a couple days ago) but I understand there has not been any significant changes since then.
Pascal, good question. We can build exclusions into the recipe so that the experiments with doorhangers don't get deployed to the same users. Also, we are aiming to launch this experiment on Monday, October 22nd. On the very slim chance that something comes up to preclude us from launching on Monday, I'd like to plan ahead for a slip to launching Wednesday, October 24. Do you (or Ritu) have any concerns about the possible date change from a release management perspective? We wouldn't launch on the 23rd due to the 63 launch.
legal is signed off on the doorhanger UX
+1 from Relman and no problem for a launch on October 24 instead of 22.
We have finalized testing the Premium Services VPN experiment Shield Study experiment. QA’s recommendation: GREEN - SHIP IT Reasoning: - All issues that were found have been fixed and verified. However, the redirect link is still the one for the Dev version of the website and not the one for the Prod version. This change is currently tracked in #72 (https://github.com/raymak/vpn-recommendation-shield-study/pull/72) Testing Summary: - Test suite: Windows 10(https://testrail.stage.mozaws.net/index.php?/runs/view/13099) & MacOS 10.13 (https://testrail.stage.mozaws.net/index.php?/runs/view/13100) Tested Platforms: - Windows 10 x64 - Mac 10.13.3 Tested Firefox versions: - Firefox Release build v62.0.3;
This version (1.0) is functionally equivalent to version 0.9 that has been approved by QA. The minimal changes are only to prepare the addon for the final deployment. Changes are limited to: - Final landing page has been set to Prod: https://premium.firefox.com/vpn/ - Telemetry testing flag has been set to "false" - The version of the addon has been set to 1.0
Devin, would you mind downloading the v1.0 and v0.9 zip files and comparing them using diff to ensure that the changes are only those that set the release parameters? I would like this to be on the record.
Hey Kamyar, I confirmed the 0.9 and 1.0 versions are the same except for the version, landing URL, and testing flag changes. ~/Downloads $ diff -r firefox_vpn_study-0.9 firefox_vpn_study-1.0 diff -r firefox_vpn_study-0.9/manifest.json firefox_vpn_study-1.0/manifest.json 4c4 < "version": "0.9", --- > "version": "1.0", diff -r firefox_vpn_study-0.9/privileged/vpnRecommender/api.js firefox_vpn_study-1.0/privileged/vpnRecommender/api.js 52c52 < const VPN_LANDING_PAGE_URL = "https://dev.psvpn.nonprod.cloudops.mozgcp.net/"; --- > const VPN_LANDING_PAGE_URL = "https://premium.firefox.com/vpn/"; diff -r firefox_vpn_study-0.9/studySetup.js firefox_vpn_study-1.0/studySetup.js 38c38 < removeTestingFlag: false, --- > removeTestingFlag: true,
We're live at 0.1% of the target population. Gradual ramp up to 0.5% expected if things look good.
Whiteboard: Github repository of the study addon
Rob: Request to move from 0.5% sample to 1% sample. We got confirmation from Jesse McCrosky on the data science team that no measurable negative impact to retention can be seen. Thanks!
We're live with the bump.
status-firefox62: affected → fixed
status-firefox63: affected → fixed
status-firefox64: affected → fixed
You need to log in before you can comment on or make changes to this bug.