Closed
Bug 1367774
Opened 7 years ago
Closed 7 years ago
Optimizely test: BATM ad campaign landing page - text version
Categories
(www.mozilla.org :: Analytics, enhancement)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: frios, Unassigned)
Details
Attachments
(1 file)
Our BATM ad campaign will run from May - August. We would will A/B test the landing pages to improve our performance through out the campaign Test audience: US, medium=display (all of our display advertising is going to this campaign) Delivery: Proposed low rotation of 10% per variation (x4) and 60% to the control. The initial page consists of text only. This is a good opportunity to determine how much information is helpful for visitors. The variations presented here include: - testing links to additional information - testing more copy on the page - prioritizing copy over the download button See details here: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+%5D+BATM+landing+page+tests+-+text+only The ad campaign has started, we would like to launch this as soon as possible.
Reporter | ||
Comment 1•7 years ago
|
||
@cmore, I need help tagging the three links on 2 variations that link off to our website. We can tag in Optimizely to measure engagement, and then tag with GA parameters for conversion. I want to make sure we can track downloads from people that click off to a different page.
Flags: needinfo?(chrismore.bugzilla)
Comment 2•7 years ago
|
||
I'll help get the experiment ready for code review.
Flags: needinfo?(chrismore.bugzilla)
Reporter | ||
Comment 3•7 years ago
|
||
(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #2) > I'll help get the experiment ready for code review. Hey Justin, thanks for cleaning it up and adding the download button and link goals. I've modified the URL targeting to only show up for people visiting from sources that start with 'y' (examples: yahoo, yh-gemini-edge-x, yh-oo-ie-age45 AND medium must always be 'display'. As long as these match, we can accept any other parameters that trail the URL. Here's the regex: ^.+www\.mozilla\.org\/en\-US\/firefox\/new\/\?xv\=batmfree\&utm\_source\=y.*\&utm\_medium=display.*$ All looks good on my side. If this checks outs, we're ready. Thanks.
Flags: needinfo?(hoosteeno)
Comment 4•7 years ago
|
||
:jpetto, can you take a look at the variation code, audience, url and goals on this experiment? I gave variation code a first pass for jquery perf, but glad to hear if there are improvements to make.
Flags: needinfo?(hoosteeno) → needinfo?(jon)
Comment 5•7 years ago
|
||
:jpetto, I think we also need optimizely enabled so it can handle requests for https://www.mozilla.org/en-US/firefox/new/?xv=batmfree
Comment 6•7 years ago
|
||
The URL targeting regex is a bit strict in that it requires the utm_ params to be in a specific order - utm_source, then utm_medium. The regex itself is fine as long as this strict ordering is acceptable. In the jQuery: - Unless I'm missing something, there's no reason to clone with data and events. Could just be: var $newCopy = $oldCopy.clone(); - As each <li> only has a single <p> child, there's no need to to specify "p:eq(X)". Could just be: $("li:eq(1) > p", $newCopy) - It's not really hurting anything, but you don't need the "/en-US" part of the injected URLs. - The last injected link in the "Inline Anchors" variation has a fully qualified destination URL. Better to keep the link relative (e.g. omit the "https://www.mozilla.org/en-US" part). Filed a PR to get Optimizely added to the ?xv=batmfree scene 1 template: https://github.com/mozilla/bedrock/pull/4864
Flags: needinfo?(jon)
Comment 7•7 years ago
|
||
(In reply to Jon Petto [:jpetto] from comment #6) > The URL targeting regex is a bit strict in that it requires the utm_ params > to be in a specific order - utm_source, then utm_medium. The regex itself is > fine as long as this strict ordering is acceptable. I asked about this too. The source of these clicks is 100% managed, so parameter ordering can be strict. > var $newCopy = $oldCopy.clone(); Fixed. > $("li:eq(1) > p", $newCopy) Fixed. > - The last injected link in the "Inline Anchors" variation has a fully > qualified destination URL. Better to keep the link relative (e.g. omit the > "https://www.mozilla.org/en-US" part). Fixed. Great comments, do you want to review again or shall we go?
Flags: needinfo?(jon)
Comment 8•7 years ago
|
||
Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/f9bfc332d50365c154e607b6d673b5b474224e55 [fix bug 1367774] Add Optimizely to /new/?xv=batmfree variation. https://github.com/mozilla/bedrock/commit/a0f26a6978d319fcb5e4aea5a8ec5b4b8b3d0a2d Merge pull request #4864 from jpetto/bug-1367774-add-optimizely-to-batmfree [fix bug 1367774] Add Optimizely to /new/?xv=batmfree variation.
Updated•7 years ago
|
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Comment 9•7 years ago
|
||
Experiment code looks good now. (Locales are still in the injected links, but they shouldn't hurt anything in this particular scenario.) The PR merged in comment 8, so all that's left is a production push and to flip the switch.
Flags: needinfo?(jon)
Comment 10•7 years ago
|
||
(In reply to Jon Petto [:jpetto] from comment #9) > Experiment code looks good now. (Locales are still in the injected links, > but they shouldn't hurt anything in this particular scenario.) > > The PR merged in comment 8, so all that's left is a production push and to > flip the switch. Thanks, jpetto! Can you update this bug after the prod push?
Reporter | ||
Comment 11•7 years ago
|
||
Thanks all. flip. the. switch! :)
Comment 12•7 years ago
|
||
The switch has been flipped and I see the Optimizely snippet on https://www.mozilla.org/en-US/firefox/new/?xv=batmfree.
Comment 13•7 years ago
|
||
David: heads up on this first easy copy/link test for the BATM page. Experiment has received all of the code reviews and it is ready to start.
Flags: needinfo?(djst)
Comment 14•7 years ago
|
||
Fabio: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes Can you update the blank results tables in the mana wiki page for each of the goals defined in the experiment recipe?
Comment 15•7 years ago
|
||
Justin: please start this experiment and note the date/time in this bug. Thanks!
Flags: needinfo?(hoosteeno)
Comment 16•7 years ago
|
||
I started the experiment this morning and immediately noticed some unfortunate visual issues caused by a conflict between the jquery code and the animation on page. I've revised the jquery in Optimizely; I hope my changes will improve the experiment. Awaiting quick dev review and then we'll launch.
Comment 17•7 years ago
|
||
Updated code looks good. r+ (Hopefully it fixes the issue...)
Comment 18•7 years ago
|
||
Experiment is running, mana page updated with start time (20170601, 12:50pm PDT). I'm watching results closely to be sure we're getting throughput.
Updated•7 years ago
|
Flags: needinfo?(hoosteeno)
Comment 19•7 years ago
|
||
A new ad has landed that is sending people to the batmprivate page, which is nearly identical to the batmfree page. We can use this experiment to handle traffic from the batmprivate ad, but we need to enable optimizely on the batmprivate template. :jpetto, can you help with that?
Flags: needinfo?(jon)
Comment 20•7 years ago
|
||
PR opened: https://github.com/mozilla/bedrock/pull/4872
Flags: needinfo?(jon)
Comment 21•7 years ago
|
||
Checked the new regex and made a couple small changes: ^.+www\.mozilla\.org\/en\-US\/firefox\/new(?:\/?).*?(?=.*?xv\=batm(?:private|free))(?=.*?utm_source\=y.*?)(?=.*?utm_medium\=display).*?$ Changes do the following: - make slash after "/new" optional - specify "utm_" before both "source" and "display" (assuming "utm_" is expected - free to remove if not) - remove superfluous ".*?" suffix on the "xv" and "utm_medium" lookaheads Regardless of the changes above, this regex is very loose. For example, the following matches: https://www.mozilla.org/en-US/firefox/new/?xv=batmfreebeer&sushiutm_source=yahoothisisfun&&&&&coffeeutm_medium=displayinawindow We can work this regex to be tighter, but it's going to get pretty complex pretty fast. (Or so I think. I am not a regex master.) If we can still be strict about parameter ordering, that would be better. Or, if we're not worried about goofy URLs like the one above, we can go with the updated regex above.
Flags: needinfo?(hoosteeno)
Comment 22•7 years ago
|
||
Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/ecc733ed0c7998d7605438d4f691a82f3c02c3b9 [fix bug 1367774] Add Optimizely to /new/?xv=batmprivate variation. - Also changes switch name for ?xv=batmfree variation https://github.com/mozilla/bedrock/commit/8438bceb56030ee88bdaed669601189377134ca5 Merge pull request #4872 from jpetto/bug-1367774-add-optimizely-to-batmprivate [fix bug 1367774] Add Optimizely to /new/?xv=batmprivate variation.
Reporter | ||
Comment 23•7 years ago
|
||
(In reply to Jon Petto [:jpetto] from comment #21) > Checked the new regex and made a couple small changes: > > ^.+www\.mozilla\.org\/en\-US\/firefox\/new(?:\/?).*?(?=.*?xv\=batm(?: > private|free))(?=.*?utm_source\=y.*?)(?=.*?utm_medium\=display).*?$ > Changes do the following: > > - make slash after "/new" optional works for me > - specify "utm_" before both "source" and "display" (assuming "utm_" is > expected - free to remove if not) we can't specify the source because it varies a bit. In some instance the source will be 'Yahoo'. Other instances it might be 'yh-gemini..', or 'yh-htpo...'. That said the medium will be always be display. > - remove superfluous ".*?" suffix on the "xv" and "utm_medium" lookaheads There are variations in the batm pages. The structure is xv-batm[something]. We want to capture all of these > Regardless of the changes above, this regex is very loose. For example, the > following matches: > > https://www.mozilla.org/en-US/firefox/new/ > ?xv=batmfreebeer&sushiutm_source=yahoothisisfun&&&&&coffeeutm_medium=displayi > nawindow that's OK. Some of those do need to be a bit loose. Of the parameters, all we need to be firm with is medium=display > We can work this regex to be tighter, but it's going to get pretty complex > pretty fast. (Or so I think. I am not a regex master.) If we can still be > strict about parameter ordering, that would be better. Or, if we're not > worried about goofy URLs like the one above, we can go with the updated > regex above. Jon, you are calling my regex goofy ;) The traffic coming to these pages is very specific. It's people clicking on one of our BATM display ads from yahoo. Fetch creates the URLs for us and they maintain this set up, in this order: https://www.mozilla.org/en-US/firefox/new/?xv=batm[something]&utm_source=[yahoo or yh-gemini&utm_medium=display&utm_campaign=[something]&utm_term=[something]&utm_content=[something]
Flags: needinfo?(jon)
Reporter | ||
Comment 24•7 years ago
|
||
The URL above looks funky, hopefully this clarifies. Here's what we need to show up for: https://www.mozilla.org/en-US/firefox/new/ ?xv=batm[suffix will vary, let's capture all possibilities such as 'batmprivate' or 'batmthisisnew'] &utm_source=[this will vary, but it will always start with a y. For example 'yahoo' or 'yh-hpto'] &utm_medium=display (will not change) &utm_campaign=[this will vary] &utm_term=[this will vary] &utm_content=[this will vary]
Comment 25•7 years ago
|
||
The regex as it stands will handle the url described in comment 24. I don't think we should block on making a perfect regex as long as it works. However, we are still blocked by not having optimizely on the page. :jpetto, can you take a look? Do we need to flip a switch? Justin
Flags: needinfo?(hoosteeno)
Comment 26•7 years ago
|
||
We are awaiting a build to complete. We will then issue a production push to get Optimizely on the page. We'll then flip the switch. Should be a couple hours, all told.
Flags: needinfo?(jon)
Comment 27•7 years ago
|
||
Commit pushed to master at https://github.com/mozmar/www-config https://github.com/mozmar/www-config/commit/b8c6595841b3ab0038a61cfcd4311318c5dd3480 [fix bug 1367774] Add Optimizely switch for /firefox/new/?xv=batmprivate. - Also renames ?xv=batmfree switch to call out Optimizely tie
Comment 28•7 years ago
|
||
Okay, the push is complete and the switches have been set. Should be ready to go on the mozilla.org side.
Comment 29•7 years ago
|
||
Experiment is live. Thanks all.
Updated•7 years ago
|
Flags: needinfo?(djst)
Reporter | ||
Comment 30•7 years ago
|
||
Hey all, just wanted to wrap up this experiment. Check out the mana for experiment summary: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes We had a winning variation which included adding additional lines of copy to the 3 features at the bottom of the page. Can you help us modify the live version by adding the additional copy to the page: Website page: https://www.mozilla.org/en-US/firefox/new/?xv=batmprivate See copy changes within the "Conclusion" section: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes
Flags: needinfo?(erenaud)
Flags: needinfo?(craigcook.bugz)
Comment 31•7 years ago
|
||
Flags: needinfo?(erenaud)
Flags: needinfo?(craigcook.bugz)
Comment 32•7 years ago
|
||
Commits pushed to master at https://github.com/mozilla/bedrock https://github.com/mozilla/bedrock/commit/728630ef14fc1215cd684e049a89f6413f476f66 [bug 1367774] Update BATM download page copy https://github.com/mozilla/bedrock/commit/0f27a1bf48da23fa1aa37db9e0976b16c10d7fdf Merge pull request #4925 from craigcook/bug-1367774-update-batm-copy [bug 1367774] Update BATM download page copy
You need to log in
before you can comment on or make changes to this bug.
Description
•