1219947 - Optimizely snippet needed for SmartOn Tracking A/B test

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, defect)

Description

[Mike Alexis [:malexis]]

Hi Jon,
Can you create an Optimizely test for our SmartOn Tracking page going live on Tues Nov 3?

We are testing showing the newsletter signup form or the Firefox download button in the footer.

https://www-demo1.allizom.org/en-US/teach/smarton/tracking/ 

I'll let Craig and Greg provide the details.

Comment 1

[Craig Cook (:craigcook)]

There are two blocks in the footer, one is class="foot-newsletter" and the other is class="foot-download" (guess which one is which). So it's just a matter of toggling display:none and display:block. 

Right now all non-Firefox browsers show the download by default so the test should just show a certain segment of those users the newsletter form instead (all Firefoxes get the newsletter).

The optimizely snippet isn't on the page at the moment but I'll be pushing an update to demo1 soon that will add it.

Comment 2

[Craig Cook (:craigcook)]

Snippet is on demo1 now and uses the waffle switch 'smarton-tracking-optimizely'

Comment 3

[Jonathan Petto [:jpetto]]

I've actually never created an Optimizely experiment - just done code review on already built experiments. Pinging cmore to see if he can put one together quickly.

Comment 4

[Chris More [:cmore]]

I was told by Greg that we aren't going to do a funnelcake for the SmartOn pages for now and we'll revisit this later. Let's needinfo him.

Comment 5

[Craig Cook (:craigcook)]

Could we still do just a basic test for the newsletter vs download? Not necessarily a full funnelcake but even just following clicks? We're planning to flip the optimizely waffle switch for this page at launch time but if there's no test we won't activate the snippet (we can always activate it later). 

So to confirm: should we NOT activate optimizely on the smarton tracking page?

Adding jgmize and ckprice for waffle awareness.

Comment 6

[Gregory Jost]

Confirming we don't need a funnelcake, but still want to a/b test CTAs for efficiency.

Comment 7

[Gregory Jost]

Hey guys,
What's the status on this? Are we live with the Optimizely logic in place?

Comment 8

[Chris More [:cmore]]

(In reply to Gregory Jost from comment #7)
> Hey guys,
> What's the status on this? Are we live with the Optimizely logic in place?

Tied up right now with other funnelcake and campaign work. 

Is this something :ckprice can get whipped up in Optimizely?

Comment 9

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Attachment: Experiment #3731003982

:cmore, :jpetto r?

Comment 10

[Jonathan Petto [:jpetto]]

Code looks good and simple to me. r+

Comment 11

[Gregory Jost]

Thanks!

Comment 12

[Chris More [:cmore]]

Checked:

* Title: updated the bug number as it was wrong

* URL: appending a / at the start of the URL

* Traffic: 100% -- that's fine since the page doesn't get massive traffic to start so we'll need all we can get

* Audience: looks good

* Goal: Looks good and capturing both the download and email sign up

r+

Comment 13

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Thanks guys - test is on.

Comment 14

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Attachment: https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=54493735

Wiki page

Comment 15

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Attachment: Experiment results

Conclusion:

FALSE - Users clicked the Download button much more often at 90% statistical significance.

No action items recommended.

Comment 16

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Greg has requested we keep this test running until Dec 7th, citing that they have new traffic sources they would like to measure the impact on.

Comment 17

[Cory Price [:ckprice] (bugmail disabled, NI me!)]

Attachment: Experiment results (Dec 8)

Updated results. Download action still a clear winner.