All users were logged out of Bugzilla on October 13th, 2018

Setup First Run Optimizely Test with FxA Value Proposition

RESOLVED FIXED

Status

RESOLVED FIXED
2 years ago
6 months ago

People

(Reporter: adavis, Unassigned)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

2 years ago
The FxA team has new ideas for first run in order to offer a better user experience by better communicating the value of Sync. Also, from user testing, we can tell that users freak out as soon as they see a sign up form and consequently don't read any value proposition.

As a first step to validating our ideas, we want to prepare an optimizely test to see if users engage more with the value proposition if they don't see the form.

We will measure the volume of accounts created and signing-in but more importantly if they are connecting a second device.

The test is being documented here:
https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=64619429

Optimizely is being setup here:
https://app.optimizely.com/edit?experiment_id=8098119674
(Reporter)

Comment 1

2 years ago
Cmore, as discussed, please review the Optimizely. It is still very much in draft but the "Preview" mode doesn't load and prevents me from testing the JQuery and JS.

Any other recommendations will be welcome.
Thanks in advance
Flags: needinfo?(chrismore.bugzilla)

Comment 2

2 years ago
BOOM! Got it working.

See running here:

https://www-demo4.allizom.org/en-US/firefox/50.0/firstrun/?test=fxa

Summary of changes:

* adding # to anchor so it doesn't make a new pageload
* wrapped div around button, because it wasn't hiding the <br> outside of the div
* changed the visibility back to visible for the fxa and masthead
* change delegate function from being triggered by the id to being triggered by the anchor tag within the fxa-intro div I created

Check it out and see what you think now.
Flags: needinfo?(chrismore.bugzilla) → needinfo?(adavis)
(Reporter)

Updated

2 years ago
Flags: needinfo?(adavis)
(Reporter)

Comment 3

2 years ago
:jpetto

Please review this optimizely test. https://app.optimizely.com/edit?experiment_id=8098119674
Test page: https://www-demo4.allizom.org/en-US/firefox/50.0/firstrun/?test=fxa

Intended behavior: 
- the Firefox accounts form is hidden by default
- An image and button are loaded instead
- when clicking on button "Setup Firefox Sync", the button and image should disappear and the form should reappear

Note:
The image is not high dpi to make sure it loads quickly. Therefore I am excluding high resolution displays from the test. (see targeting) My latest tests show that this works however I had some difficulties with this so please pay particular attention to this.

Please let me know if you have any questions.
Thanks
Flags: needinfo?(jon)
(Reporter)

Comment 4

2 years ago
Oh, and if it all looks good, please enable the Optimizely JS on prod.
(Reporter)

Comment 5

2 years ago
Small update: 
For usertesting.com users, we will be directing them to download funnelcake 97 (which is 50.1.0) since cmore is no longer using it. I will duplicate this optimizely test and run it at 100% for these funnelcake first run users so that our usertesting does not depend on the 90/10 distribution of the production a/b test.

Comment 6

2 years ago
As discussed in IRC, the high-res screen targeting was off. I've updated with the following, which seems to work:

(!window.matchMedia("(-webkit-min-device-pixel-ratio: 2), (min-device-pixel-ratio: 2), (min-resolution: 192dpi)").matches)

This line will work as long as the browsers targeted support matchMedia (which Fx does back to version 6). [1]

For more thorough testing, I've updated the code on the "Sign up Feeley" variation to point the FxA iframe to the staging server [2], which will load the iframe on demo4. After doing so, a bug was revealed - the iframe becomes visible after clicking, but has a height of 0px (due to communication from the iframe setting the height). To remedy this, I've updated the code to also set the height to 488px (which is the current loaded height of the iframe) after clicking the "Set up Firefox Sync" button. Please test this for yourselves to ensure it's working properly. When finished testing and before enabling in production, be sure to change the FxA domain back to production [3] in the variation code.

One visual bug is the stars/constellations are now laid over the terrain instead of the sky. Probably easiest to hide the stars entirely, but may be worth adjusting the CSS to keep them visible.

Also noting that the new image/copy is not responsive, so if the user happens to resize their browser below  770px wide, the image will begin to get cut off. (Probably acceptable for this test?)

Probably a good idea to do some clean up and then re-check.



[1] If you want to target browsers that might not support matchMedia (like old version of IE), I *think* the following will work:

(!window.matchMedia || !window.matchMedia("(-webkit-min-device-pixel-ratio: 2), (min-device-pixel-ratio: 2), (min-resolution: 192dpi)").matches)

However, this needs to be tested.

[2] https://accounts.stage.mozaws.net/
[3] https://accounts.firefox.com/
Flags: needinfo?(jon)
(Reporter)

Comment 7

2 years ago
Thanks for your help jpetto! 

We're only targeting Firefox users on the latest first run so we shouldn't have any IE or old Fx users.

As per the image not being responsive, it's only for this test. Chances are, if this whole test is successful, we won't be loading one giant image and will be mostly doing this with css. 

I have swaped the URLs to the prod ones.

I have duplicated the experiment for qual testing via funnelcake 97. https://app.optimizely.com/edit?experiment_id=8180153188

Please note that I duplicated version B to create a version C and slightly changed the headline because there is a hypothesis that "Your Firefox everywhere" is considered creepy to users.

cmore: please do a final double check and press play if you think it looks good.
Flags: needinfo?(chrismore.bugzilla)

Comment 8

2 years ago
Started both tests at 12:40pm PST on 1/19/2017
Flags: needinfo?(chrismore.bugzilla)
(Reporter)

Comment 9

2 years ago
thanks for your help Cmore and jpetto! I'll close this bug when we stop the test.

Comment 10

2 years ago
I see the stars/constellations are still displaying over the terrain on the page for versions B and C on both experiments. Not a great look. :/
(Reporter)

Comment 11

2 years ago
Oh snap. I can see it. It is visible in Optimizely but clearly discrete enough that none of us caught that before.

I'm making the change now.
(Reporter)

Comment 12

2 years ago
I take that back. I can't fix the stars. Cmore and I will meetup in an hour to see if we can resolve it. I'll pause the one in prod. I'll leave the user testing version up since it was already submitted.

Comment 13

2 years ago
Fixed. The easiest thing to do was to just hide the inner-container:

$(".inner-container").css({"display":"none", "visibility":""});

and then display it again later:

$(".inner-container").css({"display":"", "visibility":"visible"});

Not the most elegant thing in the world, but fine for a test for a few days to give us some insights to decide if changing the page permanently makes sense.

Comment 14

2 years ago
Updated the titles of the two Optimizely tests to note them as v1 experiments.

Copy the non-funnelcake experiment and called it v2. Removed the experiment javascript for capturing the event from the FxA iframe to trigger the Optimizely event push to eliminate that variable. I also updated the v2 to the 51.x URL format.

Here's the v2:

https://app.optimizely.com/edit?experiment_id=8193322712

Comment 15

2 years ago
Alex: let me know when you want to start v2. It is pretty much exactly the same as v1, but without the event tracking for the button click.
Flags: needinfo?(adavis)
(Reporter)

Comment 16

2 years ago
cmore: I don't have time until EOD to re-launch it.

If you have a few minutes to do that, I would appreciate it.
Flags: needinfo?(adavis)

Comment 17

2 years ago
v2 started on 1/25/2017 at 1:27pm PST.

Comment 18

6 months ago
This looks like it is complete.
Status: NEW → RESOLVED
Last Resolved: 6 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.