Closed Bug 1210492 Opened 4 years ago Closed 4 years ago

round 2 tests for windows 10 welcome page

Categories

(www.mozilla.org :: Pages & Content, defect)

Production
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jbertsch, Assigned: jpetto)

Details

(Whiteboard: [kb=1873577])

Attachments

(1 file)

44 bytes, text/x-github-pull-request
Details | Review
We propose to test the winner of our round 1 testing (v=1 here https://docs.google.com/spreadsheets/d/1HnOQpQIj0Cifs9JOzx6mN65Hw_WwZBGaivXdNpUQNQU/edit#gid=1418996668) against variations of v=1 as follows:
1.  paragraph copy removed
2.  button and image moved higher on the page to replace paragraph copy
3.  3 new headlines to be written by Matej and added to this bug

More info about future tests here:  https://etherpad.mozilla.org/welcomepage-win10-nextsteps-10ct2015
I would recommend using these three headlines:

Firefox + Windows 10. Perfect together.

Firefox works great with Windows 10

Make Firefox your default browser now
(In reply to Matej Novak [:matej] from comment #1)
> Make Firefox your default browser now

Can we make sure to keep the message about Windows 10 in all headlines? 

The goal is a combination of getting them to set the default, but also for them to know that Firefox has taken special care to work great and look great on Windows 10.

Good job on these tests! Really good to be able to see these results and effect.
(In reply to Fabio Rios [:frios] from comment #2)
> (In reply to Matej Novak [:matej] from comment #1)
> > Make Firefox your default browser now
> 
> Can we make sure to keep the message about Windows 10 in all headlines? 
> 
> The goal is a combination of getting them to set the default, but also for
> them to know that Firefox has taken special care to work great and look
> great on Windows 10.

I'm just reusing the headlines from the previous test to see which parameters affect conversion.
(In reply to Matej Novak [:matej] from comment #3)
> I'm just reusing the headlines from the previous test to see which
> parameters affect conversion.

Ah, I see that was v3 in previous test. 

Let's scrap that one for the next round of tests so we are only testing with messages that mention W10.
(In reply to Fabio Rios [:frios] from comment #4)
> (In reply to Matej Novak [:matej] from comment #3)
> > I'm just reusing the headlines from the previous test to see which
> > parameters affect conversion.
> 
> Ah, I see that was v3 in previous test. 
> 
> Let's scrap that one for the next round of tests so we are only testing with
> messages that mention W10.

I sat in on the planning meeting for this test and they are just trying to isolate all the variables from the last test. (title, body, cta, etc.) They have a good plan in place.

Mentioning that FX works great with W10 is not a goal in itself. Mentioning it serves the purpose of trying to convince user to set us as default. It's just an argument in favor. Everything comes back to getting users to set us as default. Default = >MAU.

Jenn, Chris, Gareth and Matej are looking to see what converts best to setting our browser as default. Perhaps not mentioning W10 will convert more. Nobody knows. Users never behave as expected. 

I wouldn't remove that variation from the test.
I'm updating the test plan for Round 2 of this test here:
https://mana.mozilla.org/wiki/display/FIREFOX/Windows+10+Default+CTA+Conversion+Optimization+-+Round+2

Jen, can we please ask a dev to get these 3 new variations built based on the feedback above?
Flags: needinfo?(jbertsch)
(In reply to Gareth Cull [:garethc] from comment #6)
> I'm updating the test plan for Round 2 of this test here:
> https://mana.mozilla.org/wiki/display/FIREFOX/
> Windows+10+Default+CTA+Conversion+Optimization+-+Round+2
> 
> Jen, can we please ask a dev to get these 3 new variations built based on
> the feedback above?

Hi Jon-

Do you have bandwidth to pick up this one?

Thx,
Jen
Flags: needinfo?(jbertsch) → needinfo?(jon)
(In reply to Fabio Rios [:frios] from comment #4)
> (In reply to Matej Novak [:matej] from comment #3)
> > I'm just reusing the headlines from the previous test to see which
> > parameters affect conversion.
> 
> Ah, I see that was v3 in previous test. 
> 
> Let's scrap that one for the next round of tests so we are only testing with
> messages that mention W10.

To clarify for the developer that per Fabio, this next test will test the control against 2 of the 3 proposed headlines combined with the design changes in comment 0:

Firefox + Windows 10. Perfect together.

Firefox works great with Windows 10
Thanks Jen. We wanted to make sure we delivered the Windows 10 message. But I'm going back on what I said above, go ahead and run all three. 

Let's find out the results, no harm in testing. Sorry for complicating this thing.
Yep, I've got time to take this on. I'll get started tomorrow and chime in with any questions.
Flags: needinfo?(jon)
Assignee: nobody → jon
Whiteboard: [kb=1873577]
The updated variations are on demo3:

https://www-demo3.allizom.org/en-US/firefox/windows-10/welcome/?v=1
https://www-demo3.allizom.org/en-US/firefox/windows-10/welcome/?v=2
https://www-demo3.allizom.org/en-US/firefox/windows-10/welcome/?v=3
https://www-demo3.allizom.org/en-US/firefox/windows-10/welcome/?v=4

To test, you'll need to make sure Firefox is not your default browser, and you'll need to configure Firefox to allow the UITour JS from demo3 to execute. See the instructions here: http://bedrock.readthedocs.org/en/latest/uitour.html#local-development

The value for browser.uitour.testingOrigins should have/include the URL for demo3: https://www-demo3.allizom.org
Jen and I just took a look at this. Looks great.
Excellent! Are we ready for a pull request? Or do we need to get Optimizely/other things configured before these changes go live?
(In reply to Jon Petto [:jpetto] from comment #13)
> Excellent! Are we ready for a pull request? Or do we need to get
> Optimizely/other things configured before these changes go live?

NI Gareth
Flags: needinfo?(garethcull.bugs)
Can we push the changes, then I'll create the Optimizely test to distribute the variations. Will update bug when ready. Thanks.
Flags: needinfo?(garethcull.bugs)
Attached file GitHub PR
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/5947585044a064f4f861e00bb7332341725b8eed
[fix bug 1210492] Implement round 2 of Win10 welcome testing.

https://github.com/mozilla/bedrock/commit/c8ca6c4877170d52116777711e3ca7ab1107e79e
Merge pull request #3454 from mozilla/bug-1210492-win10-welcome-tests-round2

[fix bug 1210492] Implement round 2 of Win10 welcome testing.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Hey jpetto,

Can I get a code review on this test in Optimizely please?
https://app.optimizely.com/edit?experiment_id=3723240928

Thanks.

Gareth
Flags: needinfo?(jon)
Code looks good. r+
Flags: needinfo?(jon)
The test has ran now for about 5 days and I'm going to pause it and start doing the analysis as we have sufficient sample. Thanks.
Can we now disable Optimizely on this page?
Flags: needinfo?(garethcull.bugs)
(In reply to Jon Petto [:jpetto] from comment #21)
> Can we now disable Optimizely on this page?

Gareth is going to file a bug for the next round of tests today or early next week.  So up to you if you want to turn it off and turn it back on or just leave it on.
Speeding up and securing the page, even if just for a weekend, has significant impact. Turning the switch off until the next round is ready. :)
Ok. Thanks Jon. I'll request to turn it back on when we are closer to launching round 3.
Flags: needinfo?(garethcull.bugs)
You need to log in before you can comment on or make changes to this bug.