1375936 - Optimizely test: BATM ad campaign landing page - headlines

Status: RESOLVED FIXED

(www.mozilla.org :: Analytics, enhancement)

Description

[Fabio Rios [:frios]]

We are running a headline test for our BATM landing page. This test is specific to traffic coming from our Yahoo display ad campaign.

See details here: https://mana.mozilla.org/wiki/display/FIREFOX/%5Bbug+%5D+BATM+landing+page+tests+-+headline+test

https://app.optimizely.com/edit?experiment_id=8449690745

Comment 1

[[github robot]]

Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/c71f9f358705dc432ca25b2eebe2ddbe686c5e66
[bug 1375936] Add Optimizely to batmprivate, remove traffic cop

https://github.com/mozilla/bedrock/commit/de20741104b8dd8cafea711d8ca132ccb84249f0
Merge pull request #4932 from craigcook/bug-1375936-batmprivate-optimizely

[bug 1375936] Add Optimizely to batmprivate, remove traffic cop

Comment 2

[Fabio Rios [:frios]]

Hey Justin, can you give this Optimizely test a review? 

We have 3 headline variations for the BATM landing page. Traffic is the same as the previous test, just Yahoo display ads.

https://app.optimizely.com/edit?experiment_id=8449690745

Comment 3

[[github robot]]

Commits pushed to master at https://github.com/mozmar/www-config

https://github.com/mozmar/www-config/commit/e7d0dea7c456fff95a68aed9a1e30e8921d4d77a
[bug 1375936] Activate Optimizely on batmprivate, remove Traffic Cop

https://github.com/mozmar/www-config/commit/841db77e8eb8a6f64a35296b04fca06606770981
Merge pull request #18 from craigcook/master

[bug 1375936] Activate Optimizely on batmprivate, remove Traffic Cop

Comment 4

[Justin Crawford [:hoosteeno] [:jcrawford]]

* I modified the control just to add some simulated overhead so it doesn't have a performance edge over the variations. 
* I tweaked distribution, it was doing that 24.99 vs. 25.01 thing.
* Goals, url targeting and all variation code looks good to me

:agibson, :craigcook, are either of you available for an Optimizely review?

Comment 5

[Fabio Rios [:frios]]

Sweet. Thanks

Comment 6

[Craig Cook (:craigcook)]

The setup looks good to me, but there's something odd... the URL targeting definitely specifies xv=batmprivate but the page that actually loads in the editor or the preview is xv=batmfree. This could be something cached by Optimizely since the snippet is on both pages. If you're no longer testing anything on the batmfree variant we can turn off the Optimizely snippet for that page, just to be sure.

Comment 7

[Justin Crawford [:hoosteeno] [:jcrawford]]

That's just a setting -- under "Editor Settings" you choose a URL for it to load for interactive editing, and I bet :frios has just been duplicating prior experiments so we ended up with free. I just changed it to private. 

Thanks Craig. :frios, do you want me to start this one?

Comment 8

[Craig Cook (:craigcook)]

(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #7)
> That's just a setting -- under "Editor Settings" you choose a URL for it to
> load for interactive editing, and I bet :frios has just been duplicating
> prior experiments so we ended up with free. I just changed it to private. 

Ah! I see now. Shows how infrequently I use Optimizely...

Comment 9

[Fabio Rios [:frios]]

(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #7)
> That's just a setting -- under "Editor Settings" you choose a URL for it to
> load for interactive editing, and I bet :frios has just been duplicating
> prior experiments so we ended up with free. I just changed it to private. 
> 
> Thanks Craig. :frios, do you want me to start this one?

Thank you! Yes, Justin please launch!

Comment 10

[Justin Crawford [:hoosteeno] [:jcrawford]]

This experiment started 6/26/2017.

Comment 11

[Fabio Rios [:frios]]

(In reply to Justin Crawford [:hoosteeno] [:jcrawford] from comment #10)
> This experiment started 6/26/2017.

Hey Justin, can you pause this one? We've got enough data. I'll go ahead and complete the mana page with results.

Comment 12

[Fabio Rios [:frios]]

This test is complete. There was little impact with headline variations. That said, none of the variations outperformed the control.

Complete results here: https://mana.mozilla.org/wiki/display/FIREFOX/%5B1375936%5D+BATM+landing+page+tests+-+headline+test