1310978 - A/B test engagement rates for up-to-date users on /firefox/new/

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, defect)

Description

[Alex Gibson [:agibson]]

This was first discussed in Bug 1308973 but we're splitting things out into a separate bug here so as to not over complicate things.

:jbertsch

"My goal is to see if we can improve engagement on the page as a proxy for helping existing user complete their goal in coming to the page - clearly these Firefox users are coming to the page to do something, we just don't have a great understanding of exactly what it is.  I've talked to the SUMO team and there's not a lot beyond Refresh that we offer as a "one stop fits all" fix (other than instructions on how to update manually).  And a lot of problems are related to the web pages people are visiting, not the product itself."

For the up-to-date users test:

A = the current page as control (I will ask Peter about the 'double control' here)
B = slight variation, as detailed below
C = Download button, Refresh Button, copy/link to products/ff

========================

Version B

"Congrats! You're using the latest version of Firefox."

Headline: "Browse freely"

Links (in this order):

"Reset your add-ons and settings" [https://support.mozilla.org/kb/refresh-firefox-reset-add-ons-and-settings] (Not the refresh button)

"Need more help?" [https://support.mozilla.org/products/firefox]

Button: "Free download"

========================

Version C:  

The Download Button |  The Refresh Button

"Need more help?" [https://support.mozilla.org/products/firefox]

========================

Comment 1

[Alex Gibson [:agibson]]

Attachment: GitHub pull request

Comment 2

[Alex Gibson [:agibson]]

Attachment: testsing-variants.png

Jen, given the outline above is this what you have in mind for the variants? (a/b/c left to right).

Comment 3

[Alex Gibson [:agibson]]

This is up on demo4 for testing: https://www-demo4.allizom.org/en-US/firefox/new/

Note: to see the refresh button you'll need to whitelist UITour for https://www-demo4.allizom.org/. You can do that with this handy Add-on [1] or manually via the docs [2]

[1] https://alexgibson.github.io/uitour-config/
[2] http://bedrock.readthedocs.io/en/latest/uitour.html#local-development

Comment 4

[Jennifer Bertsch [:jbertsch]]

(In reply to Alex Gibson [:agibson] from comment #2)
> Created attachment 8805913 [details]
> testsing-variants.png
> 
> Jen, given the outline above is this what you have in mind for the variants?
> (a/b/c left to right).

Thanks, Alex.  Yes.  This is what I had in mind.

Comment 5

[u570438]

Hey Jen,

Let's connect about reporting briefly tomorrow.

Thanks,

PG

Comment 6

[u570438]

I've created a data studio report that should work when this goes live. 

https://datastudio.google.com/u/0/#/org/Ra9O2QIYT8We-1pOmnS_6w/reporting/0B-vqNw0Uhi8RTVE3RkFHSzFXcFE/page/Jf7/edit

@Jen, as discussed, you'll can do the analysis when its live while I'm onboarding.
@Alex, we should have results in roughly 3 days if 100% are part of the experiment.

Comment 7

[Alex Gibson [:agibson]]

(In reply to Peter German from comment #6)
> I've created a data studio report that should work when this goes live. 
> 
> https://datastudio.google.com/u/0/#/org/Ra9O2QIYT8We-1pOmnS_6w/reporting/0B-
> vqNw0Uhi8RTVE3RkFHSzFXcFE/page/Jf7/edit
> 
> @Jen, as discussed, you'll can do the analysis when its live while I'm
> onboarding.
> @Alex, we should have results in roughly 3 days if 100% are part of the
> experiment.

Thanks Peter, will set each variation to 33%.

Comment 8

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/13184d4a932a3eefd7c6acea254e2351cefeb2c9
[fix bug 1310978] A/B test engagement rates for up-to-date users on /firefox/new/

https://github.com/mozilla/bedrock/commit/1cc89c289947053bbc1a3d8f86278121992e2b73
Merge pull request #4442 from alexgibson/bug-1310978-firefox-new-up-to-date-tests

[fix bug 1310978] A/B/C test engagement rates for up-to-date users on /firefox/new/

Comment 9

[Alex Gibson [:agibson]]

This has been pushed to prod and the experiment has been enabled.