Optimizely test: BATM ad campaign landing page - text version

RESOLVED FIXED

Status

www.mozilla.org
Analytics
RESOLVED FIXED
8 months ago
7 months ago

People

(Reporter: frios, Unassigned)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

8 months ago
Our BATM ad campaign will run from May - August. We would will A/B test the landing pages to improve our performance through out the campaign

Test audience: US, medium=display (all of our display advertising is going to this campaign)

Delivery: Proposed low rotation of 10% per variation (x4) and 60% to the control. 

The initial page consists of text only. This is a good opportunity to determine how much information is helpful for visitors. The variations presented here include:

- testing links to additional information
- testing more copy on the page
- prioritizing copy over the download button

See details here: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+%5D+BATM+landing+page+tests+-+text+only

The ad campaign has started, we would like to launch this as soon as possible.
(Reporter)

Comment 1

8 months ago
@cmore, I need help tagging the three links on 2 variations that link off to our website. We can tag in Optimizely to measure engagement, and then tag with GA parameters for conversion. I want to make sure we can track downloads from people that click off to a different page.
Flags: needinfo?(chrismore.bugzilla)
I'll help get the experiment ready for code review.
Flags: needinfo?(chrismore.bugzilla)
(Reporter)

Comment 3

8 months ago
(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #2)
> I'll help get the experiment ready for code review.

Hey Justin, thanks for cleaning it up and adding the download button and link goals. 

I've modified the URL targeting to only show up for people visiting from sources that start with 'y' (examples: yahoo, yh-gemini-edge-x, yh-oo-ie-age45 AND medium must always be 'display'. As long as these match, we can accept any other parameters that trail the URL.

Here's the regex: 

^.+www\.mozilla\.org\/en\-US\/firefox\/new\/\?xv\=batmfree\&utm\_source\=y.*\&utm\_medium=display.*$

All looks good on my side. If this checks outs, we're ready. Thanks.
Flags: needinfo?(hoosteeno)
:jpetto, can you take a look at the variation code, audience, url and goals on this experiment? I gave variation code a first pass for jquery perf, but glad to hear if there are improvements to make.
Flags: needinfo?(hoosteeno) → needinfo?(jon)
:jpetto, I think we also need optimizely enabled so it can handle requests for https://www.mozilla.org/en-US/firefox/new/?xv=batmfree

Comment 6

8 months ago
The URL targeting regex is a bit strict in that it requires the utm_ params to be in a specific order - utm_source, then utm_medium. The regex itself is fine as long as this strict ordering is acceptable.

In the jQuery:

- Unless I'm missing something, there's no reason to clone with data and events. Could just be:

var $newCopy = $oldCopy.clone();

- As each <li> only has a single <p> child, there's no need to to specify "p:eq(X)". Could just be:

$("li:eq(1) > p", $newCopy)

- It's not really hurting anything, but you don't need the "/en-US" part of the injected URLs.

- The last injected link in the "Inline Anchors" variation has a fully qualified destination URL. Better to keep the link relative (e.g. omit the "https://www.mozilla.org/en-US" part).


Filed a PR to get Optimizely added to the ?xv=batmfree scene 1 template:

https://github.com/mozilla/bedrock/pull/4864
Flags: needinfo?(jon)
(In reply to Jon Petto [:jpetto] from comment #6)
> The URL targeting regex is a bit strict in that it requires the utm_ params
> to be in a specific order - utm_source, then utm_medium. The regex itself is
> fine as long as this strict ordering is acceptable.

I asked about this too. The source of these clicks is 100% managed, so parameter ordering can be strict.

> var $newCopy = $oldCopy.clone();

Fixed.

> $("li:eq(1) > p", $newCopy)

Fixed. 

> - The last injected link in the "Inline Anchors" variation has a fully
> qualified destination URL. Better to keep the link relative (e.g. omit the
> "https://www.mozilla.org/en-US" part).

Fixed.

Great comments, do you want to review again or shall we go?
Flags: needinfo?(jon)

Comment 8

8 months ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/f9bfc332d50365c154e607b6d673b5b474224e55
[fix bug 1367774] Add Optimizely to /new/?xv=batmfree variation.

https://github.com/mozilla/bedrock/commit/a0f26a6978d319fcb5e4aea5a8ec5b4b8b3d0a2d
Merge pull request #4864 from jpetto/bug-1367774-add-optimizely-to-batmfree

[fix bug 1367774] Add Optimizely to /new/?xv=batmfree variation.

Updated

8 months ago
Status: NEW → RESOLVED
Last Resolved: 8 months ago
Resolution: --- → FIXED

Comment 9

8 months ago
Experiment code looks good now. (Locales are still in the injected links, but they shouldn't hurt anything in this particular scenario.)

The PR merged in comment 8, so all that's left is a production push and to flip the switch.
Flags: needinfo?(jon)

Comment 10

8 months ago
(In reply to Jon Petto [:jpetto] from comment #9)
> Experiment code looks good now. (Locales are still in the injected links,
> but they shouldn't hurt anything in this particular scenario.)
> 
> The PR merged in comment 8, so all that's left is a production push and to
> flip the switch.

Thanks, jpetto! Can you update this bug after the prod push?
(Reporter)

Comment 11

8 months ago
Thanks all.

flip. the. switch! :)

Comment 12

8 months ago
The switch has been flipped and I see the Optimizely snippet on https://www.mozilla.org/en-US/firefox/new/?xv=batmfree.

Comment 13

8 months ago
David: heads up on this first easy copy/link test for the BATM page. Experiment has received all of the code reviews and it is ready to start.
Flags: needinfo?(djst)

Comment 14

8 months ago
Fabio:

https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes

Can you update the blank results tables in the mana wiki page for each of the goals defined in the experiment recipe?

Comment 15

8 months ago
Justin: please start this experiment and note the date/time in this bug. Thanks!
Flags: needinfo?(hoosteeno)
I started the experiment this morning and immediately noticed some unfortunate visual issues caused by a conflict between the jquery code and the animation on page. I've revised the jquery in Optimizely; I hope my changes will improve the experiment. Awaiting quick dev review and then we'll launch.

Comment 17

8 months ago
Updated code looks good. r+

(Hopefully it fixes the issue...)
Experiment is running, mana page updated with start time (20170601, 12:50pm PDT). I'm watching results closely to be sure we're getting throughput.
Flags: needinfo?(hoosteeno)
A new ad has landed that is sending people to the batmprivate page, which is nearly identical to the batmfree page. We can use this experiment to handle traffic from the batmprivate ad, but we need to enable optimizely on the batmprivate template. :jpetto, can you help with that?
Flags: needinfo?(jon)

Comment 20

8 months ago
PR opened: https://github.com/mozilla/bedrock/pull/4872
Flags: needinfo?(jon)

Comment 21

8 months ago
Checked the new regex and made a couple small changes:

^.+www\.mozilla\.org\/en\-US\/firefox\/new(?:\/?).*?(?=.*?xv\=batm(?:private|free))(?=.*?utm_source\=y.*?)(?=.*?utm_medium\=display).*?$

Changes do the following:

- make slash after "/new" optional

- specify "utm_" before both "source" and "display" (assuming "utm_" is expected - free to remove if not)

- remove superfluous ".*?" suffix on the "xv" and "utm_medium" lookaheads

Regardless of the changes above, this regex is very loose. For example, the following matches:

https://www.mozilla.org/en-US/firefox/new/?xv=batmfreebeer&sushiutm_source=yahoothisisfun&&&&&coffeeutm_medium=displayinawindow

We can work this regex to be tighter, but it's going to get pretty complex pretty fast. (Or so I think. I am not a regex master.) If we can still be strict about parameter ordering, that would be better. Or, if we're not worried about goofy URLs like the one above, we can go with the updated regex above.
Flags: needinfo?(hoosteeno)

Comment 22

8 months ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/ecc733ed0c7998d7605438d4f691a82f3c02c3b9
[fix bug 1367774] Add Optimizely to /new/?xv=batmprivate variation.

- Also changes switch name for ?xv=batmfree variation

https://github.com/mozilla/bedrock/commit/8438bceb56030ee88bdaed669601189377134ca5
Merge pull request #4872 from jpetto/bug-1367774-add-optimizely-to-batmprivate

[fix bug 1367774] Add Optimizely to /new/?xv=batmprivate variation.
(Reporter)

Comment 23

8 months ago
(In reply to Jon Petto [:jpetto] from comment #21)
> Checked the new regex and made a couple small changes:
> 
> ^.+www\.mozilla\.org\/en\-US\/firefox\/new(?:\/?).*?(?=.*?xv\=batm(?:
> private|free))(?=.*?utm_source\=y.*?)(?=.*?utm_medium\=display).*?$
> Changes do the following:
> 
> - make slash after "/new" optional
works for me
> - specify "utm_" before both "source" and "display" (assuming "utm_" is
> expected - free to remove if not)
we can't specify the source because it varies a bit. In some instance the source will be 'Yahoo'. Other instances it might be 'yh-gemini..', or 'yh-htpo...'. That said the medium will be always be display. 
> - remove superfluous ".*?" suffix on the "xv" and "utm_medium" lookaheads
There are variations in the batm pages. The structure is xv-batm[something]. We want to capture all of these
> Regardless of the changes above, this regex is very loose. For example, the
> following matches:
> 
> https://www.mozilla.org/en-US/firefox/new/
> ?xv=batmfreebeer&sushiutm_source=yahoothisisfun&&&&&coffeeutm_medium=displayi
> nawindow
that's OK. Some of those do need to be a bit loose. Of the parameters, all we need to be firm with is medium=display
> We can work this regex to be tighter, but it's going to get pretty complex
> pretty fast. (Or so I think. I am not a regex master.) If we can still be
> strict about parameter ordering, that would be better. Or, if we're not
> worried about goofy URLs like the one above, we can go with the updated
> regex above.

Jon, you are calling my regex goofy ;)

The traffic coming to these pages is very specific. It's people clicking on one of our BATM display ads from yahoo. Fetch creates the URLs for us and they maintain this set up, in this order:

https://www.mozilla.org/en-US/firefox/new/?xv=batm[something]&utm_source=[yahoo or yh-gemini&utm_medium=display&utm_campaign=[something]&utm_term=[something]&utm_content=[something]
Flags: needinfo?(jon)
(Reporter)

Comment 24

8 months ago
The URL above looks funky, hopefully this clarifies. Here's what we need to show up for:

https://www.mozilla.org/en-US/firefox/new/

?xv=batm[suffix will vary, let's capture all possibilities such as 'batmprivate' or 'batmthisisnew']
&utm_source=[this will vary, but it will always start with a y. For example 'yahoo' or 'yh-hpto']
&utm_medium=display (will not change)
&utm_campaign=[this will vary]
&utm_term=[this will vary]
&utm_content=[this will vary]
The regex as it stands will handle the url described in comment 24. I don't think we should block on making a perfect regex as long as it works.

However, we are still blocked by not having optimizely on the page. :jpetto, can you take a look? Do we need to flip a switch?

Justin
Flags: needinfo?(hoosteeno)

Comment 26

8 months ago
We are awaiting a build to complete. We will then issue a production push to get Optimizely on the page. We'll then flip the switch. Should be a couple hours, all told.
Flags: needinfo?(jon)

Comment 27

8 months ago
Commit pushed to master at https://github.com/mozmar/www-config

https://github.com/mozmar/www-config/commit/b8c6595841b3ab0038a61cfcd4311318c5dd3480
[fix bug 1367774] Add Optimizely switch for /firefox/new/?xv=batmprivate.

- Also renames ?xv=batmfree switch to call out Optimizely tie

Comment 28

8 months ago
Okay, the push is complete and the switches have been set. Should be ready to go on the mozilla.org side.
Experiment is live. Thanks all.

Updated

7 months ago
Flags: needinfo?(djst)
(Reporter)

Comment 30

7 months ago
Hey all, just wanted to wrap up this experiment. Check out the mana for experiment summary: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes

We had a winning variation which included adding additional lines of copy to the 3 features at the bottom of the page.

Can you help us modify the live version by adding the additional copy to the page: 

Website page: https://www.mozilla.org/en-US/firefox/new/?xv=batmprivate

See copy changes within the "Conclusion" section: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+1367774%5D+BATM+landing+page+tests+-+3-value+props+copy+changes
Flags: needinfo?(erenaud)
Flags: needinfo?(craigcook.bugz)
Created attachment 8880224 [details] [review]
Github pull request https://github.com/mozilla/bedrock/pull/4925
Flags: needinfo?(erenaud)
Flags: needinfo?(craigcook.bugz)

Comment 32

7 months ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/728630ef14fc1215cd684e049a89f6413f476f66
[bug 1367774] Update BATM download page copy

https://github.com/mozilla/bedrock/commit/0f27a1bf48da23fa1aa37db9e0976b16c10d7fdf
Merge pull request #4925 from craigcook/bug-1367774-update-batm-copy

[bug 1367774] Update BATM download page copy
You need to log in before you can comment on or make changes to this bug.