1330038 - Setup First Run Optimizely Test with FxA Value Proposition

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, defect)

Description

[Alex Davis [:adavis]]

The FxA team has new ideas for first run in order to offer a better user experience by better communicating the value of Sync. Also, from user testing, we can tell that users freak out as soon as they see a sign up form and consequently don't read any value proposition.

As a first step to validating our ideas, we want to prepare an optimizely test to see if users engage more with the value proposition if they don't see the form.

We will measure the volume of accounts created and signing-in but more importantly if they are connecting a second device.

The test is being documented here:
https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=64619429

Optimizely is being setup here:
https://app.optimizely.com/edit?experiment_id=8098119674

Comment 1

[Alex Davis [:adavis]]

Cmore, as discussed, please review the Optimizely. It is still very much in draft but the "Preview" mode doesn't load and prevents me from testing the JQuery and JS.

Any other recommendations will be welcome.
Thanks in advance

Comment 2

[Chris More [:cmore]]

BOOM! Got it working.

See running here:

https://www-demo4.allizom.org/en-US/firefox/50.0/firstrun/?test=fxa

Summary of changes:

* adding # to anchor so it doesn't make a new pageload
* wrapped div around button, because it wasn't hiding the <br> outside of the div
* changed the visibility back to visible for the fxa and masthead
* change delegate function from being triggered by the id to being triggered by the anchor tag within the fxa-intro div I created

Check it out and see what you think now.

Comment 3

[Alex Davis [:adavis]]

:jpetto

Please review this optimizely test. https://app.optimizely.com/edit?experiment_id=8098119674
Test page: https://www-demo4.allizom.org/en-US/firefox/50.0/firstrun/?test=fxa

Intended behavior: 
- the Firefox accounts form is hidden by default
- An image and button are loaded instead
- when clicking on button "Setup Firefox Sync", the button and image should disappear and the form should reappear

Note:
The image is not high dpi to make sure it loads quickly. Therefore I am excluding high resolution displays from the test. (see targeting) My latest tests show that this works however I had some difficulties with this so please pay particular attention to this.

Please let me know if you have any questions.
Thanks

Comment 4

[Alex Davis [:adavis]]

Oh, and if it all looks good, please enable the Optimizely JS on prod.

Comment 5

[Alex Davis [:adavis]]

Small update: 
For usertesting.com users, we will be directing them to download funnelcake 97 (which is 50.1.0) since cmore is no longer using it. I will duplicate this optimizely test and run it at 100% for these funnelcake first run users so that our usertesting does not depend on the 90/10 distribution of the production a/b test.

Comment 6

[Jonathan Petto [:jpetto]]

As discussed in IRC, the high-res screen targeting was off. I've updated with the following, which seems to work:

(!window.matchMedia("(-webkit-min-device-pixel-ratio: 2), (min-device-pixel-ratio: 2), (min-resolution: 192dpi)").matches)

This line will work as long as the browsers targeted support matchMedia (which Fx does back to version 6). [1]

For more thorough testing, I've updated the code on the "Sign up Feeley" variation to point the FxA iframe to the staging server [2], which will load the iframe on demo4. After doing so, a bug was revealed - the iframe becomes visible after clicking, but has a height of 0px (due to communication from the iframe setting the height). To remedy this, I've updated the code to also set the height to 488px (which is the current loaded height of the iframe) after clicking the "Set up Firefox Sync" button. Please test this for yourselves to ensure it's working properly. When finished testing and before enabling in production, be sure to change the FxA domain back to production [3] in the variation code.

One visual bug is the stars/constellations are now laid over the terrain instead of the sky. Probably easiest to hide the stars entirely, but may be worth adjusting the CSS to keep them visible.

Also noting that the new image/copy is not responsive, so if the user happens to resize their browser below  770px wide, the image will begin to get cut off. (Probably acceptable for this test?)

Probably a good idea to do some clean up and then re-check.



[1] If you want to target browsers that might not support matchMedia (like old version of IE), I *think* the following will work:

(!window.matchMedia || !window.matchMedia("(-webkit-min-device-pixel-ratio: 2), (min-device-pixel-ratio: 2), (min-resolution: 192dpi)").matches)

However, this needs to be tested.

[2] https://accounts.stage.mozaws.net/
[3] https://accounts.firefox.com/

Comment 7

[Alex Davis [:adavis]]

Thanks for your help jpetto! 

We're only targeting Firefox users on the latest first run so we shouldn't have any IE or old Fx users.

As per the image not being responsive, it's only for this test. Chances are, if this whole test is successful, we won't be loading one giant image and will be mostly doing this with css. 

I have swaped the URLs to the prod ones.

I have duplicated the experiment for qual testing via funnelcake 97. https://app.optimizely.com/edit?experiment_id=8180153188

Please note that I duplicated version B to create a version C and slightly changed the headline because there is a hypothesis that "Your Firefox everywhere" is considered creepy to users.

cmore: please do a final double check and press play if you think it looks good.

Comment 8

[Chris More [:cmore]]

Started both tests at 12:40pm PST on 1/19/2017

Comment 9

[Alex Davis [:adavis]]

thanks for your help Cmore and jpetto! I'll close this bug when we stop the test.

Comment 10

[Jonathan Petto [:jpetto]]

I see the stars/constellations are still displaying over the terrain on the page for versions B and C on both experiments. Not a great look. :/

Comment 11

[Alex Davis [:adavis]]

Oh snap. I can see it. It is visible in Optimizely but clearly discrete enough that none of us caught that before.

I'm making the change now.

Comment 12

[Alex Davis [:adavis]]

I take that back. I can't fix the stars. Cmore and I will meetup in an hour to see if we can resolve it. I'll pause the one in prod. I'll leave the user testing version up since it was already submitted.

Comment 13

[Chris More [:cmore]]

Fixed. The easiest thing to do was to just hide the inner-container:

$(".inner-container").css({"display":"none", "visibility":""});

and then display it again later:

$(".inner-container").css({"display":"", "visibility":"visible"});

Not the most elegant thing in the world, but fine for a test for a few days to give us some insights to decide if changing the page permanently makes sense.

Comment 14

[Chris More [:cmore]]

Updated the titles of the two Optimizely tests to note them as v1 experiments.

Copy the non-funnelcake experiment and called it v2. Removed the experiment javascript for capturing the event from the FxA iframe to trigger the Optimizely event push to eliminate that variable. I also updated the v2 to the 51.x URL format.

Here's the v2:

https://app.optimizely.com/edit?experiment_id=8193322712

Comment 15

[Chris More [:cmore]]

Alex: let me know when you want to start v2. It is pretty much exactly the same as v1, but without the event tracking for the button click.

Comment 16

[Alex Davis [:adavis]]

cmore: I don't have time until EOD to re-launch it.

If you have a few minutes to do that, I would appreciate it.

Comment 17

[Chris More [:cmore]]

v2 started on 1/25/2017 at 1:27pm PST.

Comment 18

[pgerman]

This looks like it is complete.