Closed Bug 1219947 Opened 4 years ago Closed 4 years ago

Optimizely snippet needed for SmartOn Tracking A/B test

Categories

(www.mozilla.org :: Pages & Content, defect)

Production
defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: malexis, Assigned: ckprice)

Details

Attachments

(4 files)

Hi Jon,
Can you create an Optimizely test for our SmartOn Tracking page going live on Tues Nov 3?

We are testing showing the newsletter signup form or the Firefox download button in the footer.

https://www-demo1.allizom.org/en-US/teach/smarton/tracking/ 

I'll let Craig and Greg provide the details.
There are two blocks in the footer, one is class="foot-newsletter" and the other is class="foot-download" (guess which one is which). So it's just a matter of toggling display:none and display:block. 

Right now all non-Firefox browsers show the download by default so the test should just show a certain segment of those users the newsletter form instead (all Firefoxes get the newsletter).

The optimizely snippet isn't on the page at the moment but I'll be pushing an update to demo1 soon that will add it.
Snippet is on demo1 now and uses the waffle switch 'smarton-tracking-optimizely'
I've actually never created an Optimizely experiment - just done code review on already built experiments. Pinging cmore to see if he can put one together quickly.
Flags: needinfo?(chrismore.bugzilla)
I was told by Greg that we aren't going to do a funnelcake for the SmartOn pages for now and we'll revisit this later. Let's needinfo him.
Flags: needinfo?(chrismore.bugzilla) → needinfo?(gjost)
Could we still do just a basic test for the newsletter vs download? Not necessarily a full funnelcake but even just following clicks? We're planning to flip the optimizely waffle switch for this page at launch time but if there's no test we won't activate the snippet (we can always activate it later). 

So to confirm: should we NOT activate optimizely on the smarton tracking page?

Adding jgmize and ckprice for waffle awareness.
Confirming we don't need a funnelcake, but still want to a/b test CTAs for efficiency.
Flags: needinfo?(gjost)
Hey guys,
What's the status on this? Are we live with the Optimizely logic in place?
(In reply to Gregory Jost from comment #7)
> Hey guys,
> What's the status on this? Are we live with the Optimizely logic in place?

Tied up right now with other funnelcake and campaign work. 

Is this something :ckprice can get whipped up in Optimizely?
Flags: needinfo?(cprice)
Assignee: jon → cprice
Flags: needinfo?(cprice)
Attached file Experiment #3731003982
:cmore, :jpetto r?
Flags: needinfo?(jon)
Flags: needinfo?(chrismore.bugzilla)
Code looks good and simple to me. r+
Flags: needinfo?(jon)
Thanks!
Checked:

* Title: updated the bug number as it was wrong

* URL: appending a / at the start of the URL

* Traffic: 100% -- that's fine since the page doesn't get massive traffic to start so we'll need all we can get

* Audience: looks good

* Goal: Looks good and capturing both the download and email sign up

r+
Flags: needinfo?(chrismore.bugzilla)
Thanks guys - test is on.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Attached image Experiment results
Conclusion:

FALSE - Users clicked the Download button much more often at 90% statistical significance.

No action items recommended.
Greg has requested we keep this test running until Dec 7th, citing that they have new traffic sources they would like to measure the impact on.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Updated results. Download action still a clear winner.
Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.