Test dumb download links vs smart download button on IE 8

RESOLVED FIXED

Status

www.mozilla.org
Pages & Content
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: jpetto, Assigned: jpetto)

Tracking

Production

Firefox Tracking Flags

(Not tracked)

Details

(Assignee)

Description

2 years ago
As part of the IE <= 8 JS deprecation work (bug 1287200), we are planning to move away from the "smart" download buttons and instead only serve the no-js-required platform-direct download links.

To ensure this wont have any significant negative impact on downloads, we should run a test on IE 8 targeting specific high impact page(s).

The idea is to use Optimizely as a traffic cop (IE 8 only) and build variant pages into bedrock that force the smart download button to degrade to platform-direct links. Any issues foreseen with this approach?

The obvious page choices seem to be the home page and /firefox/new/. Are there any other pages we should test here?

Would also like recommendations on traffic size and experiment duration.
(Assignee)

Updated

2 years ago
Flags: needinfo?(jbertsch)
Flags: needinfo?(chrismore.bugzilla)
(Assignee)

Comment 1

2 years ago
Peter - Would you be able to comment here with the top 3 or 4 pages IE 8 users hit to download Firefox? That should help us decide which page(s) would be good candidates to test.
Flags: needinfo?(peter.german.bugs)
The installer help page could be a good candidate (along with /new), as it features multiple buttons:

https://www.mozilla.org/en-US/firefox/installer-help/

Comment 3

2 years ago
Traffic cop test seems to make sense. Sample size and duration depends on three things. 1) the amount of IE <=8 downloads per day 2) existing conversion rate of those downloads now 3) number of variations of the /new page you want to build. 

:garethc :pg: without dropping in IE <=8 downloads into this public bug, can you look at the current IE <=8 download run rate, assume 2 variation, look at the current rate for these users and recommend how much time we will need run the experiment for? Could just do it in an internal spreadsheet and drop it in here. 

https://www.optimizely.com/resources/sample-size-calculator/

let's assume a minimum of 5% detectable and 90% confidence.
Flags: needinfo?(chrismore.bugzilla) → needinfo?(garethcull.bugs)
(Assignee)

Comment 4

2 years ago
Noting that we are almost certainly going to be blocked running this test until bug 1282386 is resolved. The variation of /firefox/new/ (that we're likely to test) is up on a demo [1], which, when tested in IE 8, triggers the popup blocker (due to GTM). Even if Optimizely/GA count the click, I wouldn't want that segment of IE 8 users to get stuck with the popup blocker (which may well impact actual downloads/installs).

[1] https://bedrock-demo-jpetto.us-west.moz.works/en-US/firefox/new/?v=2

Comment 5

2 years ago
The landing pages responsible for downloads from IE are the following:
https://cl.ly/1P2e33450L2k

LMK if you have any questions.
Flags: needinfo?(peter.german.bugs)

Comment 6

2 years ago
(In reply to Chris More [:cmore] from comment #3)
> Traffic cop test seems to make sense. Sample size and duration depends on
> three things. 1) the amount of IE <=8 downloads per day 2) existing
> conversion rate of those downloads now 3) number of variations of the /new
> page you want to build. 
> 
> :garethc :pg: without dropping in IE <=8 downloads into this public bug, can
> you look at the current IE <=8 download run rate, assume 2 variation, look
> at the current rate for these users and recommend how much time we will need
> run the experiment for? Could just do it in an internal spreadsheet and drop
> it in here. 
> 
> https://www.optimizely.com/resources/sample-size-calculator/
> 
> let's assume a minimum of 5% detectable and 90% confidence.

You will need less than a day at a 50-50 split.
Hi Jon-

Let's just test on /firefox/new.  Thanks!
Flags: needinfo?(jbertsch)
(Assignee)

Updated

2 years ago
Depends on: 1282386
(Assignee)

Comment 8

2 years ago
Peter - 

The PR is set up to run 25% of en-US IE 8 users into the no-JS variation, and 25% into a control variation. This is easy to change prior to merging - let me know what you think.

Thanks!
Flags: needinfo?(garethcull.bugs) → needinfo?(peter.german.bugs)

Comment 9

2 years ago
Hi Jpetto,

We should be able to get results that indicate a +-2% change with two days of data 95% of the time.

If we do only one day then it'll be +-3%.

Thanks,

PG
Flags: needinfo?(peter.german.bugs)

Comment 10

2 years ago
Commits pushed to master at https://github.com/mozilla/bedrock

https://github.com/mozilla/bedrock/commit/c68c99e387983aaf4fb6c5599088e9617750dd82
[fix bug 1288517] Test dumb download links in IE <= 8.

https://github.com/mozilla/bedrock/commit/bb1fafb8679218690efd284f59ca293d5ef4c210
Merge pull request #4250 from jpetto/bug-1288517-test-ie8-dumb-download-links

[fix bug 1288517] Test dumb download links in IE <= 8.

Updated

2 years ago
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Assignee: nobody → jon
You need to log in before you can comment on or make changes to this bug.