1506301 - /new variation for Internet Explorer SEM traffic

Status: RESOLVED FIXED

(www.mozilla.org :: Pages & Content, enhancement)

Description

[kirby]

Hi! 

Requesting 4 new pages, using the existing components from the other SEM competitor landing pages. 

v=a is the control, a copy of the canonical /firefox/new page

/?xv=internetexplorer&v=a

v=1 and v=2 use this layout template: /firefox/new/?xv=opera&v=1

/?xv=internetexplorer&v=1 
/?xv=internetexplorer&v=2

v=3 and v=4 use this layout template: /firefox/new/?xv=opera&v=2

/?xv=internetexplorer&v=3
/?xv=internetexplorer&v=4

Let me know if you create a fork of /new (as with the other competitor landing page tests) and I'll redirect our Internet Explorer ads in EN to http://mozilla.org/firefox/new/?xv=internetexplorer.

The copy and the information for the comparison chart are in this document: https://docs.google.com/document/d/10jKPXbHmh7PUkId9xt4jT5QxpuPD71hCRj7HKD21dG0/edit?usp=sharing

Comment 1

[Alex Gibson [:agibson]]

Variations are now up on demo:

https://www-demo2.allizom.org/en-US/firefox/new/?xv=internetexplorer&v=a
https://www-demo2.allizom.org/en-US/firefox/new/?xv=internetexplorer&v=1
https://www-demo2.allizom.org/en-US/firefox/new/?xv=internetexplorer&v=2
https://www-demo2.allizom.org/en-US/firefox/new/?xv=internetexplorer&v=3
https://www-demo2.allizom.org/en-US/firefox/new/?xv=internetexplorer&v=4

The experiment is currently configured to allocate 20% of traffic to each cohort, but is missing utm_content values for targeting. Please provide a list of valid utm_content values for this experiment, thanks!

Any further amends, please let me know.

Comment 2

[kirby]

\o/ Thank you Alex!!! 3 things below :)

1. QA copy changes: We made updates to the copy (mostly capitalization changes and one word change). I updated the copy document, and commented all the changes so they're obvious. Sorry we didn't catch that until after we shipped it to you. https://docs.google.com/document/d/10jKPXbHmh7PUkId9xt4jT5QxpuPD71hCRj7HKD21dG0/edit?usp=sharing

2. The test will activate for the following conditions:

locale: en-US

URL params:
- utm_medium=cpc
- utm_source=google OR bing
- utm_content= [any one of the following]

A144_A634_A008782
A144_A634_C002349
A144_A634_C003778
A144_A634_A008816
A144_A634_C002352
A144_A634_C003781
A144_A203_A008845
A144_A203_C002355
A144_A203_C003768
A144_A203_C001006
A144_A203_C003771
A144_A203_A008874
A144_A203_C002358
A144_A203_C003770
A144_A634_C001302
A144_A634_C003776
A144_A634_C001283
A144_A634_C003779
A144_A203_C001035
A144_A203_C003773
A144_A203_C001055
A144_A203_C003774
A144_A203_C001045
A144_A203_C003775

3. I think for this test to work properly, we need to direct our Internet Explorer ad traffic to the root URL below. Is that correct? That's what we've done for the previous SEM landing page tests.  

http://mozilla.org/firefox/new/?xv=internetexplorer

Comment 3

[Alex Gibson [:agibson]]

(In reply to kirby from comment #2)
> \o/ Thank you Alex!!! 3 things below :)
> 
> 1. QA copy changes: We made updates to the copy (mostly capitalization
> changes and one word change). I updated the copy document, and commented all
> the changes so they're obvious. Sorry we didn't catch that until after we
> shipped it to you.
> https://docs.google.com/document/d/
> 10jKPXbHmh7PUkId9xt4jT5QxpuPD71hCRj7HKD21dG0/edit?usp=sharing
> 
> 2. The test will activate for the following conditions:
> 
> locale: en-US
> 
> URL params:
> - utm_medium=cpc
> - utm_source=google OR bing
> - utm_content= [any one of the following]
> 
> A144_A634_A008782
> A144_A634_C002349
> A144_A634_C003778
> A144_A634_A008816
> A144_A634_C002352
> A144_A634_C003781
> A144_A203_A008845
> A144_A203_C002355
> A144_A203_C003768
> A144_A203_C001006
> A144_A203_C003771
> A144_A203_A008874
> A144_A203_C002358
> A144_A203_C003770
> A144_A634_C001302
> A144_A634_C003776
> A144_A634_C001283
> A144_A634_C003779
> A144_A203_C001035
> A144_A203_C003773
> A144_A203_C001055
> A144_A203_C003774
> A144_A203_C001045
> A144_A203_C003775

Updated, thanks you. Please confirm if all looks ok, and we can move onto get this out the door :)


> 3. I think for this test to work properly, we need to direct our Internet
> Explorer ad traffic to the root URL below. Is that correct? That's what
> we've done for the previous SEM landing page tests.  
> 
> http://mozilla.org/firefox/new/?xv=internetexplorer

Correct, the URL should be https://www.mozilla.org/firefox/new/?xv=internetexplorer

Thanks

Comment 4

[kirby]

Beautiful! QA 100%. Fetch is changing the URL for all Internet Explorer ads to link to https://www.mozilla.org/firefox/new/?xv=internetexplorer

When I hear back from them, I'll update the bug with an example of a full click-through URL after it's updated.

Thanks, Alex!

Comment 5

[kirby]

Hey Alex! Here's an example of an IE SEM link. When I test it (using private windows in various browsers) I'm always seeing the control (our usual /new page). Is it working correctly? Thanks! 

https://www.mozilla.org/en-US/firefox/new/?xv=internetexplorer&utm_source=bing&utm_medium=cpc&utm_campaign=Firefox%7CBN%7CSearch%7CNB%7CExact%7CSP%7CUS%7CEN%7CDK%7CText%7CCompetitor&utm_term=internet%20explorer%20browser&utm_content=A144_A634_A008816&gclid=CJG1ltKl194CFdqFxQIdLP4Fbg&gclsrc=ds

Comment 6

[Alex Gibson [:agibson]]

(In reply to kirby from comment #5)
> Hey Alex! Here's an example of an IE SEM link. When I test it (using private
> windows in various browsers) I'm always seeing the control (our usual /new
> page). Is it working correctly? Thanks! 
> 
> https://www.mozilla.org/en-US/firefox/new/
> ?xv=internetexplorer&utm_source=bing&utm_medium=cpc&utm_campaign=Firefox%7CBN
> %7CSearch%7CNB%7CExact%7CSP%7CUS%7CEN%7CDK%7CText%7CCompetitor&utm_term=inter
> net%20explorer%20browser&utm_content=A144_A634_A008816&gclid=CJG1ltKl194CFdqF
> xQIdLP4Fbg&gclsrc=ds

This change is not yet in production (hence bug is not yet closed). Once the change is code reviewed and merged, this bug will be marked as RESOLVED -> FIXED. It will then go to production in our next scheduled push, at which point I'll update here and mark the bug as RESOLVED -> VERIFIED. Thanks.

Comment 7

[kirby]

Oooooooooooooh :) Thank you for breaking down the process for me!

Comment 8

[kirby]

Hi! What's the status?

Comment 9

[Alex Gibson [:agibson]]

(In reply to kirby from comment #8)
> Hi! What's the status?

Sorry the the delay here, the FFY campaign work took priority over this in our queue. Things are getting back to normal, so this should be reviewed and merged today. Thanks for your patience

Comment 10

[Alex Gibson [:agibson]]

Appologies for the delay in responding here, this was pushed to prod just before all-hands and then I was on PTO so am late to close this bug. Thanks for your patience.

Comment 11

[kirby]

Hi! This experiment is done. Please stop running it.
Please keep this as the winning variation: https://www.mozilla.org/en-US/firefox/new/?xv=internetexplorer&v=a

Analysis is in the SEM Landing Pages tab here: https://docs.google.com/spreadsheets/d/1UPbSeU32x8AoeHaeEHCS4Dwp4OEAvdFIVKTBqkFgloo/edit?usp=sharing

Comment 12

[kirby]

Hi! When we set up this experiment, we updated our ads to direct the appropriate traffic to this URL:
https://www.mozilla.org/firefox/new/?xv=internetexplorer

This URL should lead to the winning version, which for this experiment is the control, our usual /new page.

Thanks!