This test will compare performance of multiple landing pages for non-brand SEM ads related to "browser" keywords. Note that we are using 'content' query parameters provided by our agency in order to filter the traffic. https://app.optimizely.com/edit?experiment_id=8445162734
Hey Justin, can you review this Optimizely landing page test? I'm not sure if the download goal is set up properly to work across all variations. This AB test consists of page redirects. This test is specific to traffic coming from one of our non-brand SEM campaigns. We're using 'content' parameters supplied by Fetch in order to filter the traffic for this test. See experiment plan here: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1375967%5D+SEM+landing+page+tests+-+page+styles See Optimizely here: https://app.optimizely.com/edit?experiment_id=8445162734
Per our conversation, using the redirect in optimizely removes the query parameters and makes all traffic appear as direct. We will need to re-evaluate options to determine how to complete this test.
Justin, Fetch confirmed they include the query parameters for all this traffic. I'd like to give it a try. Can we launch? I'll keep an eye on it this week. Here's the test: https://app.optimizely.com/edit?experiment_id=8445162734
I reviewed this and made 2 modifications: 1) I built a new goal that tracks clicks either on the download button on www, or on the download links on blog. 2) I added an audience (it was defaulting to "all") -- any browser, en-US desktop. :jpetto, are you available for an Optimizely review? Links above.
Flags: needinfo?(hoosteeno) → needinfo?(jon)
I can likely review this later today, or early tomorrow.
(In reply to Jon Petto [:jpetto] from comment #5) > I can likely review this later today, or early tomorrow. Thank you! Let's launch as soon as ready!
Code for the variations looks good (the redirects), but the targeting URL regex needs some work. As-is, it will currently match: https://www.mozilla.org/en-US/firefox/new/?xv=workingout&utm_source=google I think the "or" conditions need to be contained in square brackets to not impact the rest of the regex. The URL targeting match validator in the Optimizely interface is nice for testing. Also noting that we'll need to open a PR to get Optimizely added to the /new?xv=workingout variation. I will file this PR this afternoon.
I added parens around the alternating terms (square brackets work for chars) to fix the bug :jpetto identified. And because any 358-character regex that works is both priceless and irreplaceable, I'm copying it here too. ^.+www\.mozilla\.org\/en\-US\/firefox\/new\/\?xv\=workingout\&utm\_source\=(google|bing)\&utm\_medium=cpc\&utm\_campaign\=.*?\&utm\_term\=.*?\&utm\_content\=(A144\_A203\_A007848|A144\_A203\_A007835|A144\_A203\_C000998|A144\_A634\_A008779|A144\_A634\_A008813|A144\_A203\_A007850|A144\_A203\_A007837|A144\_A203\_C000994|A144\_A634\_A008778|A144\_A634\_A008812)
Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/34e5710c3da41d36c9db1dd30983cca415ea62bd [fix bug 1375967] Add Optimizely to Fx new lifestyle workingout variation. https://github.com/mozilla/bedrock/commit/3d6b49bb54da256c47c5e79ca7cf3b76de251cbd Merge pull request #4975 from jpetto/bug-1375967-enable-optimizely-fxnew-workingout-variant [fix bug 1375967] Add Optimizely to Fx new lifestyle workingout variant
Status: NEW → RESOLVED
Last Resolved: 10 months ago
Resolution: --- → FIXED
Commits pushed to master at https://github.com/mozmar/www-config https://github.com/mozmar/www-config/commit/30cbbf77bd8f5b3925ebc419f32b1528b0f93ded [fix bug 1375967] Enable switch for fxlifestyle workingout optimizely https://github.com/mozmar/www-config/commit/17ee47ba592b9ee59b55a927dbba210b271de73c Merge pull request #23 from jpetto/bug-1375967-fxlifestyle-workingout-optimizely [fix bug 1375967] Enable switch for fxlifestyle workingout optimizely
Regex looks good now. Optimizely will be enabled on prod in about an hour (just waiting on tests to run prior to push).
(In reply to Jon Petto [:jpetto] from comment #11) > Regex looks good now. Optimizely will be enabled on prod in about an hour > (just waiting on tests to run prior to push). Thank you! Can we activate the test today?
Optimizely is now present on the template in production. Will let Justin handle starting the experiment.
I have launched the experiment. I had to make a small change to code in /new variation, since redirecting to /new while retaining params is eqivalent to landing on the control. My change was to replace xv=workingout with xv=x1 in params. Fabio, please keep an eye on it and ping me if it needs changing over the weekend or after.
I also need to point out a late-breaking realization: This test is redirecting to multiple pages and subdomains and they aren't all set up to use optimizely. For that reason, we're going to need to track goals using GA. We can do that by focusing on sessions that... * land on one of the pages we're redirecting to * in desktop en-US * with the very specific utm_content params we're targeting
(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #15) > I also need to point out a late-breaking realization: This test is > redirecting to multiple pages and subdomains and they aren't all set up to > use optimizely. For that reason, we're going to need to track goals using > GA. We can do that by focusing on sessions that... > > * land on one of the pages we're redirecting to > * in desktop en-US > * with the very specific utm_content params we're targeting I can't seem to find these in GA. However it looks like we might be able to pull numbers from optimizely using either scene 2 metrics, or download button click numbers. Let me know what you think.
> I can't seem to find these in GA. Look in landing pages; add a second dimension, "Ad Content", and filter by regex on the utm_content parameters coming with these ads. You can also filter on page URL if you want. > > However it looks like we might be able to pull numbers from optimizely using > either scene 2 metrics, or download button click numbers. I've made some adjustments to the experiment (and reset data). I hope we'll be able to use the "viewed scene2" as a global measure of conversion regardless of which page they landed on. Let's watch.
:jpetto, can you add optimizely to the scene=2 pages for xv=workingout and xv=batmprivate? That should fix the optimizely tracking issues we've seen.
Commits pushed to master at https://github.com/mozmeao/www-config https://github.com/mozmeao/www-config/commit/3d058cdc022f2769d3e28f15e577932974977672 [fix bug 1375967] Enable optimizely fxnew batm scene 2. https://github.com/mozmeao/www-config/commit/8ea2f375d1c91f5c4163f968018d123ae5a22410 Merge pull request #26 from jpetto/bug-1375967-enable-optimizely-batm-scene2 [fix bug 1375967] Enable optimizely fxnew batm scene 2.
Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/99cb5cdf6061cb50303e1a828722c200580291ce [fix bug 1375967] Add Optimizely to scene 2 on /new variations: - batm* - workingout https://github.com/mozilla/bedrock/commit/7ad6706965e9820f434c4b2d1b90ae524b13983b Merge pull request #4994 from jpetto/bug-1375967-add-optimizely-to-batmprivate-workingout-fxnew-variations [fix bug 1375967] Add Optimizely to scene 2 on /new variations:
You need to log in before you can comment on or make changes to this bug.