1219379 - Optimizely Test: /whatsnew 42 version A and B

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, defect)

Description

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

This bug is opened to create an Optimizely test for two versions of /whatsnew.

Traffic
10% of total en-US/whatsnew traffic

Variations
A 50% https://www-demo2.allizom.org/en-US/firefox/42.0/whatsnew/
B 50% https://www-demo2.allizom.org/en-US/firefox/42.0/whatsnew/b/

KPI
Clicks on the Try it button

Audience
Firefox, English

Responsibilities
ckprice - create Optimizely test
growth - review in Optimizely
frontend dev - code review

Launch
Targeting Nov 5

Comment 1

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Attachment: Optimizely Test: /firefox/42.0/whatsnew Tracking Protection Layout Test (German)

This test is created in Optimizely.

:cmore r?

Comment 2

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Attachment: Wiki Page

Comment 3

[Chris More [:cmore]]

Checked:

* URL target: Adjusted the URL target to exclude /de/firefox/42.0/whatsnew/b/ as it would create a potential redirect loop that after a user is redirected to /b/ that they could fall back into the experiment and get redirected again.

* Audience: Looks fine

* Goal: Adjusted the URL target for the goal to be simply "/de/firefox/42.0/whatsnew/" so it captures the original and variation due to the change above with the URL target.

* Traffic allocation: The 10% looks fine

Comment 4

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Thanks Chris - I'll offer it to :jpetto to review, but it may be short since it's all just redirecting traffic around.

Comment 5

[Jonathan Petto [:jpetto]]

Code looks good to me. r+

Comment 6

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Chatted with cmore, I'm going to start this today.

Comment 7

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Attachment: Experiment results

I'm going to end this test.

Results are attached.

The conclusion here is that a simple format (headline, graphic, button) for the /whatsnew page (and possibly other pages) is more effective at users clicking on the button.

Comment 8

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Closing bug - I will also update the Wiki.

Comment 9

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

jbertsch/habber - I'm pretty satisfied with the results here. Thoughts? Anything else you'd like to test with version B? Otherwise we'll probably remove it from bedrock

Comment 10

[Jennifer Bertsch [:jbertsch]]

(In reply to Cory Price [:ckprice] from comment #9)
> jbertsch/habber - I'm pretty satisfied with the results here. Thoughts?
> Anything else you'd like to test with version B? Otherwise we'll probably
> remove it from bedrock

Yep.  Turn it off and remove B from Bedrock.  Thanks, Cory.