Closed Bug 1367774 Opened 7 years ago Closed 7 years ago

Optimizely test: BATM ad campaign landing page - text version

Categories

(www.mozilla.org :: Analytics, enhancement)

Production
enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: frios, Unassigned)

Details

Attachments

(1 file)

Our BATM ad campaign will run from May - August. We would will A/B test the landing pages to improve our performance through out the campaign

Test audience: US, medium=display (all of our display advertising is going to this campaign)

Delivery: Proposed low rotation of 10% per variation (x4) and 60% to the control. 

The initial page consists of text only. This is a good opportunity to determine how much information is helpful for visitors. The variations presented here include:

- testing links to additional information
- testing more copy on the page
- prioritizing copy over the download button

See details here: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+%5D+BATM+landing+page+tests+-+text+only

The ad campaign has started, we would like to launch this as soon as possible.
@cmore, I need help tagging the three links on 2 variations that link off to our website. We can tag in Optimizely to measure engagement, and then tag with GA parameters for conversion. I want to make sure we can track downloads from people that click off to a different page.
Flags: needinfo?(chrismore.bugzilla)
I'll help get the experiment ready for code review.
Flags: needinfo?(chrismore.bugzilla)
(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #2)
> I'll help get the experiment ready for code review.

Hey Justin, thanks for cleaning it up and adding the download button and link goals. 

I've modified the URL targeting to only show up for people visiting from sources that start with 'y' (examples: yahoo, yh-gemini-edge-x, yh-oo-ie-age45 AND medium must always be 'display'. As long as these match, we can accept any other parameters that trail the URL.

Here's the regex: 

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/\?xv\=batmfree\&utm\_source\=y.*\&utm\_medium=display.*$

All looks good on my side. If this checks outs, we're ready. Thanks.
Flags: needinfo?(hoosteeno)
:jpetto, can you take a look at the variation code, audience, url and goals on this experiment? I gave variation code a first pass for jquery perf, but glad to hear if there are improvements to make.
Flags: needinfo?(hoosteeno) → needinfo?(jon)
:jpetto, I think we also need optimizely enabled so it can handle requests for https://www.mozilla.org/en-US/firefox/new/?xv=batmfree
The URL targeting regex is a bit strict in that it requires the utm_ params to be in a specific order - utm_source, then utm_medium. The regex itself is fine as long as this strict ordering is acceptable.

In the jQuery:

- Unless I'm missing something, there's no reason to clone with data and events. Could just be:

var $newCopy = $oldCopy.clone();

- As each <li> only has a single <p> child, there's no need to to specify "p:eq(X)". Could just be:

$("li:eq(1) > p", $newCopy)

- It's not really hurting anything, but you don't need the "/en-US" part of the injected URLs.

- The last injected link in the "Inline Anchors" variation has a fully qualified destination URL. Better to keep the link relative (e.g. omit the "https://www.mozilla.org/en-US" part).


Filed a PR to get Optimizely added to the ?xv=batmfree scene 1 template:

https://github.com/mozilla/bedrock/pull/4864
Flags: needinfo?(jon)
(In reply to Jon Petto [:jpetto] from comment #6)
> The URL targeting regex is a bit strict in that it requires the utm_ params
> to be in a specific order - utm_source, then utm_medium. The regex itself is
> fine as long as this strict ordering is acceptable.

I asked about this too. The source of these clicks is 100% managed, so parameter ordering can be strict.

> var $newCopy = $oldCopy.clone();

Fixed.

> $("li:eq(1) > p", $newCopy)

Fixed. 

> - The last injected link in the "Inline Anchors" variation has a fully
> qualified destination URL. Better to keep the link relative (e.g. omit the
> "https://www.mozilla.org/en-US" part).

Fixed.

Great comments, do you want to review again or shall we go?
Flags: needinfo?(jon)
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/f9bfc332d50365c154e607b6d673b5b474224e55
[fix bug 1367774] Add Optimizely to /new/?xv=batmfree variation.

https://github.com/mozilla/bedrock/commit/a0f26a6978d319fcb5e4aea5a8ec5b4b8b3d0a2d
Merge pull request #4864 from jpetto/bug-1367774-add-optimizely-to-batmfree

[fix bug 1367774] Add Optimizely to /new/?xv=batmfree variation.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Experiment code looks good now. (Locales are still in the injected links, but they shouldn't hurt anything in this particular scenario.)

The PR merged in comment 8, so all that's left is a production push and to flip the switch.
Flags: needinfo?(jon)
(In reply to Jon Petto [:jpetto] from comment #9)
> Experiment code looks good now. (Locales are still in the injected links,
> but they shouldn't hurt anything in this particular scenario.)
> 
> The PR merged in comment 8, so all that's left is a production push and to
> flip the switch.

Thanks, jpetto! Can you update this bug after the prod push?
Thanks all.

flip. the. switch! :)
The switch has been flipped and I see the Optimizely snippet on https://www.mozilla.org/en-US/firefox/new/?xv=batmfree.
David: heads up on this first easy copy/link test for the BATM page. Experiment has received all of the code reviews and it is ready to start.
Flags: needinfo?(djst)
Fabio:

https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes

Can you update the blank results tables in the mana wiki page for each of the goals defined in the experiment recipe?
Justin: please start this experiment and note the date/time in this bug. Thanks!
Flags: needinfo?(hoosteeno)
I started the experiment this morning and immediately noticed some unfortunate visual issues caused by a conflict between the jquery code and the animation on page. I've revised the jquery in Optimizely; I hope my changes will improve the experiment. Awaiting quick dev review and then we'll launch.
Updated code looks good. r+

(Hopefully it fixes the issue...)
Experiment is running, mana page updated with start time (20170601, 12:50pm PDT). I'm watching results closely to be sure we're getting throughput.
Flags: needinfo?(hoosteeno)
A new ad has landed that is sending people to the batmprivate page, which is nearly identical to the batmfree page. We can use this experiment to handle traffic from the batmprivate ad, but we need to enable optimizely on the batmprivate template. :jpetto, can you help with that?
Flags: needinfo?(jon)
Checked the new regex and made a couple small changes:

^.+www\.mozilla\.org\/en\-US\/firefox\/new(?:\/?).*?(?=.*?xv\=batm(?:private|free))(?=.*?utm_source\=y.*?)(?=.*?utm_medium\=display).*?$

Changes do the following:

- make slash after "/new" optional

- specify "utm_" before both "source" and "display" (assuming "utm_" is expected - free to remove if not)

- remove superfluous ".*?" suffix on the "xv" and "utm_medium" lookaheads

Regardless of the changes above, this regex is very loose. For example, the following matches:

https://www.mozilla.org/en-US/firefox/new/?xv=batmfreebeer&sushiutm_source=yahoothisisfun&&&&&coffeeutm_medium=displayinawindow

We can work this regex to be tighter, but it's going to get pretty complex pretty fast. (Or so I think. I am not a regex master.) If we can still be strict about parameter ordering, that would be better. Or, if we're not worried about goofy URLs like the one above, we can go with the updated regex above.
Flags: needinfo?(hoosteeno)
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/ecc733ed0c7998d7605438d4f691a82f3c02c3b9
[fix bug 1367774] Add Optimizely to /new/?xv=batmprivate variation.

- Also changes switch name for ?xv=batmfree variation

https://github.com/mozilla/bedrock/commit/8438bceb56030ee88bdaed669601189377134ca5
Merge pull request #4872 from jpetto/bug-1367774-add-optimizely-to-batmprivate

[fix bug 1367774] Add Optimizely to /new/?xv=batmprivate variation.
(In reply to Jon Petto [:jpetto] from comment #21)
> Checked the new regex and made a couple small changes:
> 
> ^.+www\.mozilla\.org\/en\-US\/firefox\/new(?:\/?).*?(?=.*?xv\=batm(?:
> private|free))(?=.*?utm_source\=y.*?)(?=.*?utm_medium\=display).*?$
> Changes do the following:
> 
> - make slash after "/new" optional
works for me
> - specify "utm_" before both "source" and "display" (assuming "utm_" is
> expected - free to remove if not)
we can't specify the source because it varies a bit. In some instance the source will be 'Yahoo'. Other instances it might be 'yh-gemini..', or 'yh-htpo...'. That said the medium will be always be display. 
> - remove superfluous ".*?" suffix on the "xv" and "utm_medium" lookaheads
There are variations in the batm pages. The structure is xv-batm[something]. We want to capture all of these
> Regardless of the changes above, this regex is very loose. For example, the
> following matches:
> 
> https://www.mozilla.org/en-US/firefox/new/
> ?xv=batmfreebeer&sushiutm_source=yahoothisisfun&&&&&coffeeutm_medium=displayi
> nawindow
that's OK. Some of those do need to be a bit loose. Of the parameters, all we need to be firm with is medium=display
> We can work this regex to be tighter, but it's going to get pretty complex
> pretty fast. (Or so I think. I am not a regex master.) If we can still be
> strict about parameter ordering, that would be better. Or, if we're not
> worried about goofy URLs like the one above, we can go with the updated
> regex above.

Jon, you are calling my regex goofy ;)

The traffic coming to these pages is very specific. It's people clicking on one of our BATM display ads from yahoo. Fetch creates the URLs for us and they maintain this set up, in this order:

https://www.mozilla.org/en-US/firefox/new/?xv=batm[something]&utm_source=[yahoo or yh-gemini&utm_medium=display&utm_campaign=[something]&utm_term=[something]&utm_content=[something]
Flags: needinfo?(jon)
The URL above looks funky, hopefully this clarifies. Here's what we need to show up for:

https://www.mozilla.org/en-US/firefox/new/

?xv=batm[suffix will vary, let's capture all possibilities such as 'batmprivate' or 'batmthisisnew']
&utm_source=[this will vary, but it will always start with a y. For example 'yahoo' or 'yh-hpto']
&utm_medium=display (will not change)
&utm_campaign=[this will vary]
&utm_term=[this will vary]
&utm_content=[this will vary]
The regex as it stands will handle the url described in comment 24. I don't think we should block on making a perfect regex as long as it works.

However, we are still blocked by not having optimizely on the page. :jpetto, can you take a look? Do we need to flip a switch?

Justin
Flags: needinfo?(hoosteeno)
We are awaiting a build to complete. We will then issue a production push to get Optimizely on the page. We'll then flip the switch. Should be a couple hours, all told.
Flags: needinfo?(jon)
Commit pushed to master at https://github.com/mozmar/www-config

https://github.com/mozmar/www-config/commit/b8c6595841b3ab0038a61cfcd4311318c5dd3480
[fix bug 1367774] Add Optimizely switch for /firefox/new/?xv=batmprivate.

- Also renames ?xv=batmfree switch to call out Optimizely tie
Okay, the push is complete and the switches have been set. Should be ready to go on the mozilla.org side.
Experiment is live. Thanks all.
Flags: needinfo?(djst)
Hey all, just wanted to wrap up this experiment. Check out the mana for experiment summary: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes

We had a winning variation which included adding additional lines of copy to the 3 features at the bottom of the page.

Can you help us modify the live version by adding the additional copy to the page: 

Website page: https://www.mozilla.org/en-US/firefox/new/?xv=batmprivate

See copy changes within the "Conclusion" section: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes
Flags: needinfo?(erenaud)
Flags: needinfo?(craigcook.bugz)
Flags: needinfo?(erenaud)
Flags: needinfo?(craigcook.bugz)
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/728630ef14fc1215cd684e049a89f6413f476f66
[bug 1367774] Update BATM download page copy

https://github.com/mozilla/bedrock/commit/0f27a1bf48da23fa1aa37db9e0976b16c10d7fdf
Merge pull request #4925 from craigcook/bug-1367774-update-batm-copy

[bug 1367774] Update BATM download page copy
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: