1375967 - Optimizely test: SEM landing page test - page variations

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, enhancement)

Description

[Fabio Rios [:frios]]

This test will compare performance of multiple landing pages for non-brand SEM ads related to "browser" keywords.

Note that we are using 'content' query parameters provided by our agency in order to filter the traffic.

https://app.optimizely.com/edit?experiment_id=8445162734

Comment 1

[Fabio Rios [:frios]]

Hey Justin, can you review this Optimizely landing page test? I'm not sure if the download goal is set up properly to work across all variations.

This AB test consists of page redirects. This test is specific to traffic coming from one of our non-brand SEM campaigns. We're using 'content' parameters supplied by Fetch in order to filter the traffic for this test.

See experiment plan here: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1375967%5D+SEM+landing+page+tests+-+page+styles
See Optimizely here: https://app.optimizely.com/edit?experiment_id=8445162734

Comment 2

[Fabio Rios [:frios]]

Per our conversation, using the redirect in optimizely removes the query parameters and makes all traffic appear as direct. We will need to re-evaluate options to determine how to complete this test.

Comment 3

[Fabio Rios [:frios]]

Justin, 

Fetch confirmed they include the query parameters for all this traffic. I'd like to give it a try. Can we launch? I'll keep an eye on it this week.

Here's the test: https://app.optimizely.com/edit?experiment_id=8445162734

Comment 4

[Justin Crawford [:hoosteeno] [:jcrawford]]

I reviewed this and made 2 modifications:

1) I built a new goal that tracks clicks either on the download button on www, or on the download links on blog. 
2) I added an audience (it was defaulting to "all") -- any browser, en-US desktop.

:jpetto, are you available for an Optimizely review? Links above.

Comment 5

[Jonathan Petto [:jpetto]]

I can likely review this later today, or early tomorrow.

Comment 6

[Fabio Rios [:frios]]

(In reply to Jon Petto [:jpetto] from comment #5)
> I can likely review this later today, or early tomorrow.

Thank you! Let's launch as soon as ready!

Comment 7

[Jonathan Petto [:jpetto]]

Code for the variations looks good (the redirects), but the targeting URL regex needs some work. As-is, it will currently match:

https://www.mozilla.org/en-US/firefox/new/?xv=workingout&utm_source=google

I think the "or" conditions need to be contained in square brackets to not impact the rest of the regex. The URL targeting match validator in the Optimizely interface is nice for testing.

Also noting that we'll need to open a PR to get Optimizely added to the /new?xv=workingout variation. I will file this PR this afternoon.

Comment 8

[Justin Crawford [:hoosteeno] [:jcrawford]]

I added parens around the alternating terms (square brackets work for chars) to fix the bug :jpetto identified. And because any 358-character regex that works is both priceless and irreplaceable, I'm copying it here too.

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/\?xv\=workingout\&utm\_source\=(google|bing)\&utm\_medium=cpc\&utm\_campaign\=.*?\&utm\_term\=.*?\&utm\_content\=(A144\_A203\_A007848|A144\_A203\_A007835|A144\_A203\_C000998|A144\_A634\_A008779|A144\_A634\_A008813|A144\_A203\_A007850|A144\_A203\_A007837|A144\_A203\_C000994|A144\_A634\_A008778|A144\_A634\_A008812)

Comment 9

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/34e5710c3da41d36c9db1dd30983cca415ea62bd
[fix bug 1375967] Add Optimizely to Fx new lifestyle workingout variation.

https://github.com/mozilla/bedrock/commit/3d6b49bb54da256c47c5e79ca7cf3b76de251cbd
Merge pull request #4975 from jpetto/bug-1375967-enable-optimizely-fxnew-workingout-variant

[fix bug 1375967] Add Optimizely to Fx new lifestyle workingout variant

Comment 10

[[github robot]]

Commits pushed to master at https://github.com/mozmar/www-config

https://github.com/mozmar/www-config/commit/30cbbf77bd8f5b3925ebc419f32b1528b0f93ded
[fix bug 1375967] Enable switch for fxlifestyle workingout optimizely

https://github.com/mozmar/www-config/commit/17ee47ba592b9ee59b55a927dbba210b271de73c
Merge pull request #23 from jpetto/bug-1375967-fxlifestyle-workingout-optimizely

[fix bug 1375967] Enable switch for fxlifestyle workingout optimizely

Comment 11

[Jonathan Petto [:jpetto]]

Regex looks good now. Optimizely will be enabled on prod in about an hour (just waiting on tests to run prior to push).

Comment 12

[Fabio Rios [:frios]]

(In reply to Jon Petto [:jpetto] from comment #11)
> Regex looks good now. Optimizely will be enabled on prod in about an hour
> (just waiting on tests to run prior to push).

Thank you! Can we activate the test today?

Comment 13

[Jonathan Petto [:jpetto]]

Optimizely is now present on the template in production. Will let Justin handle starting the experiment.

Comment 14

[Justin Crawford [:hoosteeno] [:jcrawford]]

I have launched the experiment. I had to make a small change to code in /new variation, since redirecting to /new while retaining params is eqivalent to landing on the control. My change was to replace xv=workingout with xv=x1 in params.

Fabio, please keep an eye on it and ping me if it needs changing over the weekend or after.

Comment 15

[Justin Crawford [:hoosteeno] [:jcrawford]]

I also need to point out a late-breaking realization: This test is redirecting to multiple pages and subdomains and they aren't all set up to use optimizely. For that reason, we're going to need to track goals using GA. We can do that by focusing on sessions that...

* land on one of the pages we're redirecting to
* in desktop en-US
* with the very specific utm_content params we're targeting

Comment 16

[Fabio Rios [:frios]]

(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #15)
> I also need to point out a late-breaking realization: This test is
> redirecting to multiple pages and subdomains and they aren't all set up to
> use optimizely. For that reason, we're going to need to track goals using
> GA. We can do that by focusing on sessions that...
> 
> * land on one of the pages we're redirecting to
> * in desktop en-US
> * with the very specific utm_content params we're targeting

I can't seem to find these in GA.

However it looks like we might be able to pull numbers from optimizely using either scene 2 metrics, or download button click numbers.

Let me know what you think.

Comment 17

[Justin Crawford [:hoosteeno] [:jcrawford]]

> I can't seem to find these in GA.

Look in landing pages; add a second dimension, "Ad Content", and filter by regex on the utm_content parameters coming with these ads. You can also filter on page URL if you want.

> 
> However it looks like we might be able to pull numbers from optimizely using
> either scene 2 metrics, or download button click numbers.

I've made some adjustments to the experiment (and reset data). I hope we'll be able to use the "viewed scene2" as a global measure of conversion regardless of which page they landed on. Let's watch.

Comment 18

[Justin Crawford [:hoosteeno] [:jcrawford]]

:jpetto, can you add optimizely to the scene=2 pages for xv=workingout and xv=batmprivate? That should fix the optimizely tracking issues we've seen.

Comment 19

[Jonathan Petto [:jpetto]]

Attachment: GitHub PRs

Comment 20

[[github robot]]

Commits pushed to master at https://github.com/mozmeao/www-config

https://github.com/mozmeao/www-config/commit/3d058cdc022f2769d3e28f15e577932974977672
[fix bug 1375967] Enable optimizely fxnew batm scene 2.

https://github.com/mozmeao/www-config/commit/8ea2f375d1c91f5c4163f968018d123ae5a22410
Merge pull request #26 from jpetto/bug-1375967-enable-optimizely-batm-scene2

[fix bug 1375967] Enable optimizely fxnew batm scene 2.

Comment 21

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/99cb5cdf6061cb50303e1a828722c200580291ce
[fix bug 1375967] Add Optimizely to scene 2 on /new variations:

- batm*
- workingout

https://github.com/mozilla/bedrock/commit/7ad6706965e9820f434c4b2d1b90ae524b13983b
Merge pull request #4994 from jpetto/bug-1375967-add-optimizely-to-batmprivate-workingout-fxnew-variations

[fix bug 1375967] Add Optimizely to scene 2 on /new variations: