Closed Bug 901028 Opened 11 years ago Closed 11 years ago

Phase 2: First Run testing: ADI test

Categories

(www.mozilla.org :: General, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: cmore, Assigned: jpetto)

References

Details

(Whiteboard: [kb=1101931] )

This bug will be used to define the requirements and methods to do an ADI test on how /firstrun content influences long-term usage of the Firefox Desktop product.
Depends on: 900658
Blocks: 901026
No longer depends on: 901026
No longer depends on: 853869
Here is an idea on how we can do a real A/B test with funnelcake and firstrun content. Content will vary on the firstrun pages by the funnelcake build ID in the URL since that is now passed through to buttons and the end product. Google content experiments is used to do the 80/10/10 split (20% traffic) to guide the visitor to the normal and two funnelcake builds of Firefox. This is en-US only, but will need to work for Windows, Mac, and Linux. 

1) Create the final new /firstrun content and host it at:

https://www.mozilla.org/en-US/firefox/23.0/firstrun/?f=25 (content will vary depending on query parameter existing and equal to 25)

2) Allow the current /firstrun page to be accessible via either of these URLs:

https://www.mozilla.org/en-US/firefox/22.0/firstrun/
https://www.mozilla.org/en-US/firefox/22.0/firstrun/f?=24

3) Create a GA experiment with these URLs:

https://www.mozilla.org/en-US/firefox/new/ (original) 80%
https://www.mozilla.org/en-US/firefox/new/?f=24 (variant A -- funnelcake control) %10
https://www.mozilla.org/en-US/firefox/new/?f=25 (variant B -- new content) %10

The en-US version /new page will either redirect to the same URL with f=24 or f=25. While the original and f=25 are the same content, the control needs to be a funnelcake so that we can monitor the A/B test through the entire funnel. No additional bedrock changes need to be done for the /new outside of just adding the experiment code since the SEM campaign allows us to vary the download button for funnelcake testing by default now.

===

Here are the 3 user flows:

1) Visitor views https://www.mozilla.org/en-US/firefox/new/, downloads normal Firefox build, installs Firefox, opens Firefox, pops https://www.mozilla.org/en-US/firefox/23.0/firstrun/ tab, see *original* firstrun content.

2) Visitor views https://www.mozilla.org/en-US/firefox/new/, gets redirected to: https://www.mozilla.org/en-US/firefox/new/?f=24, downloads Firefox Funnelcake 24 (control), installs Firefox, opens Firefox, pops https://www.mozilla.org/en-US/firefox/23.0/firstrun/?f=24 tab, sees *original* firstrun content.

3) Visitor views https://www.mozilla.org/en-US/firefox/new/, gets redirected to: https://www.mozilla.org/en-US/firefox/new/?f=25, downloads Firefox Funnelcake 25 (variant), installs Firefox, opens Firefox, pops https://www.mozilla.org/en-US/firefox/23.0/firstrun/?f=25 tab, see *new* firstrun content.
Gareth/Anurag: Can you do a sanity check on comment 1?
Flags: needinfo?(garethcull.bugs)
Flags: needinfo?(aphadke)
Chris,

Just to clarify, we want to send 20% of visitors to the /new page into the Experiment using GA. GA will then split this traffic equally among the original and 2 variants (?f=24, ?f=25) and send them to a /new page as mentioned in Comment 1. 

Gareth
Flags: needinfo?(garethcull.bugs)
(In reply to Gareth Cull [:garethc] from comment #3)
> Chris,
> 
> Just to clarify, we want to send 20% of visitors to the /new page into the
> Experiment using GA. GA will then split this traffic equally among the
> original and 2 variants (?f=24, ?f=25) and send them to a /new page as
> mentioned in Comment 1. 
> 
> Gareth

Yes, it is an 80/10/10 split and with GA, that would be 20%. 80% of people would get nothing different than they do now, 10% of people would get f=24 and 10% of people would get f=25. When users make it to the /firstrun page, if there is no f=number, they get the current experience, if f=24 then they still get the current experience, and if f=25, they get the new updated content. All the URLs will be the same for the firstrun except for the f=whatever query parameters.
I did a sanity check with Anurag on on the A/B test in comment 1 and he said it looks fine. We can do this post-SEM test.
Flags: needinfo?(aphadke)
Hey Chris,

GA will only let us send 25% of traffic into the experiment. There is no option to send 20% in the drop down menu.

You can review the experiment setting here: 

https://www.google.com/analytics/web/?et&authuser#siteopt-experiment/siteopt-experiment-wizard/a36116321w64087799p71153379/%3F_u.date00%3D20130708%26_u.date01%3D20130821%26createExperimentWizard.experimentId%3DvG1F80OiT9mXu6FaOITmDg/

Here's the GA Experiments code that is to be placed after the opening <head> tag that is to be placed on the Original page: 
https://www.mozilla.org/en-US/firefox/new/

Also, not sure if you will need to suppress this experiments code for the variation urls as it is essentially the same link as the original, but with a query parameter appended. 

<!-- Google Analytics Content Experiment code -->
<script>function utmx_section(){}function utmx(){}(function(){var
k='71153379-28',d=document,l=d.location,c=d.cookie;
if(l.search.indexOf('utm_expid='+k)>0)return;
function f(n){if(c){var i=c.indexOf(n+'=');if(i>-1){var j=c.
indexOf(';',i);return escape(c.substring(i+n.length+1,j<0?c.
length:j))}}}var x=f('__utmx'),xx=f('__utmxx'),h=l.hash;d.write(
'<sc'+'ript src="'+'http'+(l.protocol=='https:'?'s://ssl':
'://www')+'.google-analytics.com/ga_exp.js?'+'utmxkey='+k+
'&utmx='+(x?x:'')+'&utmxx='+(xx?xx:'')+'&utmxtime='+new Date().
valueOf()+(h?'&utmxhash='+escape(h.substr(1)):'')+
'" type="text/javascript" charset="utf-8"><\/sc'+'ript>')})();
</script><script>utmx('url','A/B');</script>
<!-- End of Google Analytics Content Experiment code -->
25% is fine. It will be a 75/12.5/12.5 experiment. Thanks!

Developer note: We should wrap the experiment code in a conditional that will only be displayed in the head *if* the lang is en-US.
Also, please note that the play event on the original firstrun page is broken. Can we also fix this? 

https://bugzilla.mozilla.org/show_bug.cgi?id=908883
Depends on: 908883
(In reply to Gareth Cull [:garethc] from comment #8)
> Also, please note that the play event on the original firstrun page is
> broken. Can we also fix this? 
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=908883

Yes, we should resolve this first for a true A/B test.
Hi Chris and Mike-

What are the next steps to get this live?  Do you think we could have it up and running by the end of the week?

Thx,
Jen
Flags: needinfo?(malexis)
Flags: needinfo?(chrismore.bugzilla)
(In reply to Jennifer Bertsch [:jbertsch] from comment #10)
> Hi Chris and Mike-
> 
> What are the next steps to get this live?  Do you think we could have it up
> and running by the end of the week?
> 
> Thx,
> Jen

Next step is to get jpetto to build the new page described in bug 909943. After that is live on mozilla.org and the GA tag is on /firefox/new/, we can get the funnelcake builds created. I just want everything in place so that we are not too close to a 6-week Firefox cycle.
Flags: needinfo?(chrismore.bugzilla)
Flags: needinfo?(malexis)
Whiteboard: [kb=1101931]
Assignee: nobody → jon
Commit pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/b257564c4bf1ba56bf6128216aa29b13351f4b78
Add querystring test URL for fx firstrun. Bugs 901028, 909943.

- Remove old A/B firstrun test URLs

- Remove B firstrun template

- Remove GA experiment code from /firefox/firstrun/

- Add GA experiment code to /firefox/new/
verified the following on stage:

GA experiment code is on https://www.allizom.org/en-US/firefox/new/
The experiment code removed from https://www.allizom.org/en-US/firefox/firstrun/
The A/B test pages have been removed from stage.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
I've verified the code is here:

https://www.mozilla.org/en-US/firefox/new/

and not here:

https://www.mozilla.org/es-ES/firefox/new/

Thanks!
Status: RESOLVED → VERIFIED
Raymond:

Can you verify all these two URLs and six variations for Windows, Mac, and Linux? Test these are non-Firefox latest browsers. Each download should have either -f24 or -f25 in the d.m.o. link and not 404.

http://www.mozilla.org/en-US/firefox/new/?f=24

* Windows:
* Mac:
* Linux:

http://www.mozilla.org/en-US/firefox/new/?f=25

* Windows
* Mac:
* Linux:

This is for en-US only. Other locales will still return en-US.
Status: VERIFIED → REOPENED
Flags: needinfo?(mozbugs.retornam)
Resolution: FIXED → ---
http://www.mozilla.org/en-US/firefox/new/?f=24:
Mac:
   Chrome : https://download.mozilla.org/?product=firefox-24.0-f24&os=osx&lang=en-US
   Firefox 23:
   Safari:https://download.mozilla.org/?product=firefox-24.0-f24&os=osx&lang=en-US
   Opera: https://download.mozilla.org/?product=firefox-24.0-f24&os=osx&lang=en-US

Windows:
   Chrome: https://download.mozilla.org/?product=firefox-stub-f24&os=win&lang=en-US
   Firefox 23: https://download.mozilla.org/?product=firefox-stub-f24&os=win&lang=en-US
   Safari:https://download.mozilla.org/?product=firefox-stub-f24&os=win&lang=en-US
   Internet Explorer: https://download.mozilla.org/?product=firefox-stub-f24&os=win&lang=en-US
   

Download works and there is no 404.

http://www.mozilla.org/en-US/firefox/new/?f=25
Mac: 
  Chrome: https://download.mozilla.org/?product=firefox-24.0-f25&os=osx&lang=en-US
  Firefox 23:https://download.mozilla.org/?product=firefox-24.0-f25&os=osx&lang=en-US
  Safari: https://download.mozilla.org/?product=firefox-24.0-f25&os=osx&lang=en-US
  Opera:https://download.mozilla.org/?product=firefox-24.0-f25&os=osx&lang=en-US

Windows:
   Chrome: https://download.mozilla.org/?product=firefox-stub-f25&os=win&lang=en-US
   Firefox 23: https://download.mozilla.org/?product=firefox-stub-f25&os=win&lang=en-US
   Safari:https://download.mozilla.org/?product=firefox-stub-f25&os=win&lang=en-US
   Internet Explorer: https://download.mozilla.org/?product=firefox-stub-f25&os=win&lang=en-US

Download works and there is no 404.
We are live on production and Gareth and I have verified the GA experiment is looking good. We are going to run for 2 weeks.
Status: REOPENED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → FIXED
This test was stopped today and we will have to wait 30, 60, 90 days to see the impact on ADIs.
You need to log in before you can comment on or make changes to this bug.