1248728 - Optimizely test: /firefox/new/ separated templates

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, defect)

Description

[Jonathan Petto [:jpetto]]

Need to create and run an Optimizely experiment to ensure new /firefox/new/ templates do not have a negative impact on conversion.

New templates use only pre-localized strings, so experiment does not need to be locale restricted.

Comment 1

[Jonathan Petto [:jpetto]]

I've created a first draft of the experiment [1].

Questions for cmore & garethc:

- What percent of visitors should we send into this experiment?

- Should we limit to certain locales, or allow all users?

- Do we need to do anything on the template side in terms of GA/GTM to ensure we can properly track conversions? (I believe garethc already answered "no" to this, but want to double check.)

For reference, the updated /firefox/new/ page can be accessed at https://www.mozilla.org/en-US/firefox/new/?v=2. Note that nothing should be visibly different - it's just a simpler/lighter template.


[1] https://app.optimizely.com/projects/246059135/experiments/5013527311

Comment 2

[Chris More [:cmore]]

(In reply to Jon Petto [:jpetto] from comment #1)
> I've created a first draft of the experiment [1].
> 
> Questions for cmore & garethc:
> 
> - What percent of visitors should we send into this experiment?

Given the high volume already and that we are touching the top of the funnel, I would start with 5% and watch results closely after you start it.
> 
> - Should we limit to certain locales, or allow all users?

If you are not changing any of the strings and it should work in all locales, then you should just target all languages. I would just change it to non-Firefox visitors, which there is an existing audience for. If you want the cleanest conversion rate, you could do english + organic visitors as they have the most consistent and highest conversion rate, but targeting all locales/languages should be fine.

I added the non-Firefox audience, unless you are fine with targeting Firefox visitors that may or may not have a download button. You can remove it if you want.

> 
> - Do we need to do anything on the template side in terms of GA/GTM to
> ensure we can properly track conversions? (I believe garethc already
> answered "no" to this, but want to double check.)
> 

I double checked myself and I can see the real-time GA events coming through on the v=2, so it looks like we are fine.

> For reference, the updated /firefox/new/ page can be accessed at
> https://www.mozilla.org/en-US/firefox/new/?v=2. Note that nothing should be
> visibly different - it's just a simpler/lighter template.

Sounds good.

I also added a goal to track downloads for each of the variations. Even though you'll be doing it in GA, it is nice to get the numbers quickly in Optimizely so you don't have to wait 3 hours to see if the experiment is running. If you look at the goal, it is just a selector and I adjusted the goal targeted to be something different than the experiment url (just a sub-string of /firefox/new/ so that it goal can match all of the test v=X variations. If you don't change the goal target, the goal will only work on the experiment URL and not the v=2 one.

> 
> 
> [1] https://app.optimizely.com/projects/246059135/experiments/5013527311

Comment 3

[Jonathan Petto [:jpetto]]

Thanks for the excellent feedback and experiment updates :cmore!

Comment 4

[Jonathan Petto [:jpetto]]

After about 2 hours and ~3600 visitors, I'm seeing a > 90% drop in conversions for both the Engagement & Firefox Download goals. I've paused the test until we can figure out what's going on.

:cmore & :garethc - Can you take a look when you have time?

Comment 5

[Chris More [:cmore]]

(In reply to Jon Petto [:jpetto] from comment #4)
> After about 2 hours and ~3600 visitors, I'm seeing a > 90% drop in
> conversions for both the Engagement & Firefox Download goals. I've paused
> the test until we can figure out what's going on.
> 
> :cmore & :garethc - Can you take a look when you have time?

I have an idea.

You know how we had issues with GA events with the second scene and the download save-is that interrupted JS? I wonder if that is what is happening here that the load of the second page and then the save-is is blocking Optimizely from being able to tell someone clicked on the .download-content button? Do you have any delays in there before loading the second page like we do with the non-separate templates?

Comment 6

[Chris More [:cmore]]

Actually... I see what it is... you don't have Optimizely enabled on the v=2 template! :)

Comment 7

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/dfde76ffe5482265dbcb0ad8eac75cbc597166de
[fix bug 1248728] New /new templates - update waffle switch names

- Should match current /new template waffle switch name
  during A/B testing

https://github.com/mozilla/bedrock/commit/2cf723e5be50be0b33bcc695b4c4936e2f56cd92
Merge pull request #3883 from jpetto/bug-1248728-fx-new-update-waffle-switches

[fix bug 1248728] New /new templates - update waffle switch names

Comment 8

[Gareth Cull [:garethc]]

Clearing NI