Optimizely test: SEM landing page test - page variations

RESOLVED FIXED

Status

www.mozilla.org
Analytics
RESOLVED FIXED
11 months ago
10 months ago

People

(Reporter: frios, Unassigned)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

11 months ago
This test will compare performance of multiple landing pages for non-brand SEM ads related to "browser" keywords.

Note that we are using 'content' query parameters provided by our agency in order to filter the traffic.

https://app.optimizely.com/edit?experiment_id=8445162734
(Reporter)

Comment 1

11 months ago
Hey Justin, can you review this Optimizely landing page test? I'm not sure if the download goal is set up properly to work across all variations.

This AB test consists of page redirects. This test is specific to traffic coming from one of our non-brand SEM campaigns. We're using 'content' parameters supplied by Fetch in order to filter the traffic for this test.

See experiment plan here: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1375967%5D+SEM+landing+page+tests+-+page+styles
See Optimizely here: https://app.optimizely.com/edit?experiment_id=8445162734
Flags: needinfo?(hoosteeno)
(Reporter)

Comment 2

11 months ago
Per our conversation, using the redirect in optimizely removes the query parameters and makes all traffic appear as direct. We will need to re-evaluate options to determine how to complete this test.
(Reporter)

Comment 3

11 months ago
Justin, 

Fetch confirmed they include the query parameters for all this traffic. I'd like to give it a try. Can we launch? I'll keep an eye on it this week.

Here's the test: https://app.optimizely.com/edit?experiment_id=8445162734
I reviewed this and made 2 modifications:

1) I built a new goal that tracks clicks either on the download button on www, or on the download links on blog. 
2) I added an audience (it was defaulting to "all") -- any browser, en-US desktop.

:jpetto, are you available for an Optimizely review? Links above.
Flags: needinfo?(hoosteeno) → needinfo?(jon)

Comment 5

10 months ago
I can likely review this later today, or early tomorrow.
Flags: needinfo?(jon)
(Reporter)

Comment 6

10 months ago
(In reply to Jon Petto [:jpetto] from comment #5)
> I can likely review this later today, or early tomorrow.

Thank you! Let's launch as soon as ready!

Comment 7

10 months ago
Code for the variations looks good (the redirects), but the targeting URL regex needs some work. As-is, it will currently match:

https://www.mozilla.org/en-US/firefox/new/?xv=workingout&utm_source=google

I think the "or" conditions need to be contained in square brackets to not impact the rest of the regex. The URL targeting match validator in the Optimizely interface is nice for testing.

Also noting that we'll need to open a PR to get Optimizely added to the /new?xv=workingout variation. I will file this PR this afternoon.
I added parens around the alternating terms (square brackets work for chars) to fix the bug :jpetto identified. And because any 358-character regex that works is both priceless and irreplaceable, I'm copying it here too.

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/\?xv\=workingout\&utm\_source\=(google|bing)\&utm\_medium=cpc\&utm\_campaign\=.*?\&utm\_term\=.*?\&utm\_content\=(A144\_A203\_A007848|A144\_A203\_A007835|A144\_A203\_C000998|A144\_A634\_A008779|A144\_A634\_A008813|A144\_A203\_A007850|A144\_A203\_A007837|A144\_A203\_C000994|A144\_A634\_A008778|A144\_A634\_A008812)

Comment 9

10 months ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/34e5710c3da41d36c9db1dd30983cca415ea62bd
[fix bug 1375967] Add Optimizely to Fx new lifestyle workingout variation.

https://github.com/mozilla/bedrock/commit/3d6b49bb54da256c47c5e79ca7cf3b76de251cbd
Merge pull request #4975 from jpetto/bug-1375967-enable-optimizely-fxnew-workingout-variant

[fix bug 1375967] Add Optimizely to Fx new lifestyle workingout variant

Updated

10 months ago
Status: NEW → RESOLVED
Last Resolved: 10 months ago
Resolution: --- → FIXED

Comment 10

10 months ago
Commits pushed to master at https://github.com/mozmar/www-config

https://github.com/mozmar/www-config/commit/30cbbf77bd8f5b3925ebc419f32b1528b0f93ded
[fix bug 1375967] Enable switch for fxlifestyle workingout optimizely

https://github.com/mozmar/www-config/commit/17ee47ba592b9ee59b55a927dbba210b271de73c
Merge pull request #23 from jpetto/bug-1375967-fxlifestyle-workingout-optimizely

[fix bug 1375967] Enable switch for fxlifestyle workingout optimizely

Comment 11

10 months ago
Regex looks good now. Optimizely will be enabled on prod in about an hour (just waiting on tests to run prior to push).
(Reporter)

Comment 12

10 months ago
(In reply to Jon Petto [:jpetto] from comment #11)
> Regex looks good now. Optimizely will be enabled on prod in about an hour
> (just waiting on tests to run prior to push).

Thank you! Can we activate the test today?
Flags: needinfo?(jon)
Flags: needinfo?(hoosteeno)

Comment 13

10 months ago
Optimizely is now present on the template in production. Will let Justin handle starting the experiment.
Flags: needinfo?(jon)
I have launched the experiment. I had to make a small change to code in /new variation, since redirecting to /new while retaining params is eqivalent to landing on the control. My change was to replace xv=workingout with xv=x1 in params.

Fabio, please keep an eye on it and ping me if it needs changing over the weekend or after.
Flags: needinfo?(hoosteeno)
I also need to point out a late-breaking realization: This test is redirecting to multiple pages and subdomains and they aren't all set up to use optimizely. For that reason, we're going to need to track goals using GA. We can do that by focusing on sessions that...

* land on one of the pages we're redirecting to
* in desktop en-US
* with the very specific utm_content params we're targeting
(Reporter)

Comment 16

10 months ago
(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #15)
> I also need to point out a late-breaking realization: This test is
> redirecting to multiple pages and subdomains and they aren't all set up to
> use optimizely. For that reason, we're going to need to track goals using
> GA. We can do that by focusing on sessions that...
> 
> * land on one of the pages we're redirecting to
> * in desktop en-US
> * with the very specific utm_content params we're targeting

I can't seem to find these in GA.

However it looks like we might be able to pull numbers from optimizely using either scene 2 metrics, or download button click numbers.

Let me know what you think.
> I can't seem to find these in GA.

Look in landing pages; add a second dimension, "Ad Content", and filter by regex on the utm_content parameters coming with these ads. You can also filter on page URL if you want.

> 
> However it looks like we might be able to pull numbers from optimizely using
> either scene 2 metrics, or download button click numbers.

I've made some adjustments to the experiment (and reset data). I hope we'll be able to use the "viewed scene2" as a global measure of conversion regardless of which page they landed on. Let's watch.
:jpetto, can you add optimizely to the scene=2 pages for xv=workingout and xv=batmprivate? That should fix the optimizely tracking issues we've seen.
Flags: needinfo?(jon)

Comment 19

10 months ago
Created attachment 8889543 [details]
GitHub PRs
Flags: needinfo?(jon)

Comment 20

10 months ago
Commits pushed to master at https://github.com/mozmeao/www-config

https://github.com/mozmeao/www-config/commit/3d058cdc022f2769d3e28f15e577932974977672
[fix bug 1375967] Enable optimizely fxnew batm scene 2.

https://github.com/mozmeao/www-config/commit/8ea2f375d1c91f5c4163f968018d123ae5a22410
Merge pull request #26 from jpetto/bug-1375967-enable-optimizely-batm-scene2

[fix bug 1375967] Enable optimizely fxnew batm scene 2.

Comment 21

10 months ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/99cb5cdf6061cb50303e1a828722c200580291ce
[fix bug 1375967] Add Optimizely to scene 2 on /new variations:

- batm*
- workingout

https://github.com/mozilla/bedrock/commit/7ad6706965e9820f434c4b2d1b90ae524b13983b
Merge pull request #4994 from jpetto/bug-1375967-add-optimizely-to-batmprivate-workingout-fxnew-variations

[fix bug 1375967] Add Optimizely to scene 2 on /new variations:
You need to log in before you can comment on or make changes to this bug.