1256241 - Implement /new download page redesign tests

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, defect)

Description

[Alex Gibson [:agibson]]

Wire frames: http://cl.ly/0L1L2D3e0H3P

Download page copy: https://docs.google.com/document/d/12aaIzS1HD5qtMnM6dBXKyHUiwR5h7Nx6Tpn5yBe5vC4/edit

Thank you page copy: https://docs.google.com/document/d/1iqxVBrnrUu8ZoQye-Gp5ZzPDW4pelwZuP_9WprUgOhA/edit

Download page v1 (rocket only):
https://www.dropbox.com/s/s8mikyhj95o0mss/firefox-space_download.psd?dl=0

Download page v2 (rocket and browser chrome):
https://www.dropbox.com/s/5ku2aak1ltlo9og/firefox-space_download_v2.psd?dl=0

Comment 1

[Alex Gibson [:agibson]]

Content strategy & test plan: https://docs.google.com/document/d/1Po7pohV58qJARzcowCMdiLoUcSkmbq4OKBtWAy6pzME/edit#heading=h.nu1prtz844tl

Onboarding web pages testing: https://docs.google.com/presentation/d/1QC48ZmfnnrEbz-T2mSNN6x8PEhHJAZ-NlKZbLzof0UY/edit#slide=id.g11dd08bf7e_3_2

Comment 2

[Alex Gibson [:agibson]]

Revised ground-level contour, as per the discussion today: https://www.dropbox.com/s/s8mikyhj95o0mss/firefox-space_download.psd?dl=0

Comment 3

[Alex Gibson [:agibson]]

Attachment: browser-chrome_2x.psd

Browser chrome high-res asset

Comment 4

[Alex Gibson [:agibson]]

Attachment: rocket_2x.psd

Rocket high-res asset

Comment 5

[Alex Gibson [:agibson]]

Download Thank You:
https://www.dropbox.com/s/1voyxftbv84njdi/firefox-space_download-interstitial.psd?dl=0


Cheering Foxes (2x):
https://www.dropbox.com/s/hp83xdsej2y4lmg/foxes-on-hill_2x.psd?dl=0


Rocket in Space (2x):
https://www.dropbox.com/s/gdij8jui7jsic2c/rocket-in-space_2x.psd?dl=0

Comment 6

[Alex Gibson [:agibson]]

OK, this is still WIP but variations are now up on demo for general feedback:

Download Page Variations
------------------------

https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=a
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=b
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=c
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=d
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=e
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=f
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=g
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=h

Thank You Page Variations
-------------------------

https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?scene=2&v=a
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?scene=2&v=b

Comment 7

[Alex Gibson [:agibson]]

I should also add, probably best to use Chrome to view these URLs ^

Comment 8

[Gareth Cull [:garethc]]

Thanks Agibson. 

Could we change the variation lettering, from a-h to something like 1a, 1b to 4a, 4b where a and b is the scene 2 variation that the user sees. It would make the reporting easier. Here's a screenshot as to how i've set it up so far in Data Studio: cl.ly/2T1h2E0a1P1d

Comment 9

[Chris More [:cmore]]

(In reply to Gareth Cull [:garethc] from comment #8)
> Thanks Agibson. 
> 
> Could we change the variation lettering, from a-h to something like 1a, 1b
> to 4a, 4b where a and b is the scene 2 variation that the user sees. It
> would make the reporting easier. Here's a screenshot as to how i've set it
> up so far in Data Studio: cl.ly/2T1h2E0a1P1d

I am fine with letters or numbers as long as we are consistent. Probably numbers for the first scene and letters for the second scene would be easier to figure out since it is more unique.

1a
1b
1c
1d
2a
2b
2c
2d

Should we just use no parameters for the controls or should we have a specific variation for the control?

We will have to setup the first scene to randomly include the a & b variation URLs for the second scene.

Comment 10

[Alex Gibson [:agibson]]

Since we have 4 variations of scene1 and 2 variations of scene2, I think Gareth's suggestion makes sense:

1a, 2a, 3a, 4a
1b, 2b, 3b, 4b

The template variations already link through to the appropriate version of scene2, we can do this all in bedrock.

Comment 11

[Alex Gibson [:agibson]]

URLs are now as follows as per Gareth's suggestion:

Download Page Variations
------------------------

https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=1a
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=2a
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=3a
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=4a

https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=1b
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=2b
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=3b
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?v=4b

Thank You Page Variations
-------------------------

https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?scene=2&v=a
https://bedrock-demo-agibson.us-west.moz.works/en-US/firefox/new/?scene=2&v=b

Gareth, does this work better for you?

Note I haven't included a control version via a parameter. If this is an issue please let me know.

Comment 12

[Gareth Cull [:garethc]]

Thanks Alex. This looks good.

Comment 13

[Alex Gibson [:agibson]]

Attachment: GitHub pull request

Comment 14

[Chris More [:cmore]]

Looks good! Nice job

Comment 15

[Gareth Cull [:garethc]]

Alex,

Will these get pushed to a demo server so I can test our Analytics?

Comment 16

[Alex Gibson [:agibson]]

(In reply to Gareth Cull [:garethc] from comment #15)
> Alex,
> 
> Will these get pushed to a demo server so I can test our Analytics?

Gareth, it's on one of our new demo server environments, please see comment 11 ^

Comment 17

[Chris More [:cmore]]

:agibson: Is there a bug to perform the optimizely test and if not, who has the action to get the bug and test setup? I want to get it rolling asap. Want to validate the new design is on par with the existing design.

Comment 18

[Chris More [:cmore]]

:agibson: another question. How much did you compress the graphics on the page (where you didn't use css) knowing that page weight can impact conversion rates on low bandwidth?

Comment 19

[Alex Gibson [:agibson]]

(In reply to Chris More [:cmore] from comment #17)
> :agibson: Is there a bug to perform the optimizely test and if not, who has
> the action to get the bug and test setup? I want to get it rolling asap.
> Want to validate the new design is on par with the existing design.

Chris, it was my understanding that you had agreed to take on setting up the Optimizely test? (bug 1256762). Not sure where wires have got crossed here, but if this is not the case I'll consult with Jen.

The images have been compressed as much as possible without degrading them. There is obviously going to be some differentiation in page weight, since the design is a lot more image heavy. No amount of image-smushing can get around this everywhere. We have used CSS where ever possible.

Ideally, we should have really used a performance budget for this page during the design process (something I have been pushing for over the past few years).

Comment 20

[Alex Gibson [:agibson]]

Chris, this has been given an r+ in code review - any reason not to merge/push this change?

Comment 21

[Chris More [:cmore]]

(In reply to Alex Gibson [:agibson] from comment #20)
> Chris, this has been given an r+ in code review - any reason not to
> merge/push this change?

I am fine with getting this merged and pushed and we'll get the Optimizely test setup and needinfo you guys to review the test when ready before anything is started. Verdi also want to do some user testing with these URLs, so he'll wait until its on production. 

Thanks!

Comment 22

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/bc20f6b190b3652380d96855c6407afe590bc8cc
[fix bug 1256241] Implement /new download page redesign tests

https://github.com/mozilla/bedrock/commit/3a408810bc1b975a60a5704b8033a4a986888e4c
Merge pull request #3992 from alexgibson/bug-1256241-download-pages-redesign-tests

[fix bug 1256241] Implement /new download page redesign tests

Comment 23

[Chris More [:cmore]]

Looks like we are on prod now:

Download Page Variations
------------------------

https://www.mozilla.org/en-US/firefox/new/?v=1a
https://www.mozilla.org/en-US/firefox/new/?v=2a
https://www.mozilla.org/en-US/firefox/new/?v=3a
https://www.mozilla.org/en-US/firefox/new/?v=4a

https://www.mozilla.org/en-US/firefox/new/?v=1b
https://www.mozilla.org/en-US/firefox/new/?v=2b
https://www.mozilla.org/en-US/firefox/new/?v=3b
https://www.mozilla.org/en-US/firefox/new/?v=4b

Thank You Page Variations
-------------------------

https://www.mozilla.org/en-US/firefox/new/?scene=2&v=a
https://www.mozilla.org/en-US/firefox/new/?scene=2&v=b

Comment 24

[Chris More [:cmore]]

Results of tests in test details page: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=60588271

Conclusion: Page performs as good as the control or original page in primary traffic mediums.