1199441 - Understand the impact on suggested tiles on Firefox desktop retention

Status: RESOLVED WONTFIX

(Content Services Graveyard :: General, defect)

Description

[Chris More [:cmore]]

The Firefox Growth, Platform, and Content Services teams would like to perform multiple cohort tests of Firefox desktop users to understand how suggested tiles impacts core product metrics.

We will start out with a simple test and then based off of the learning, we'll design the next test.

== Test 1 Proposal ==

Hypothesis:

User with suggested tiles enabled will be retained as long as users without suggested tiles enabled.

Approach:

For an upcoming version of Firefox desktop, use funnelcake builds to create cohorts with different Firefox settings enabled.

Segment:

Channel: release
Version: 41.0 (or another future version)
Lang: en-US
OS: win32

Cohort 1: Full installer build of Firefox with no changes that includes sponsored tiles.

Cohort 2: Full installer build of Firefox with suggested tiles disabled (via config change when the funnelcake is built)

Goals: 

* Understand any churn/retention over time by week for each cohort (we can do this already with funnelcakes)

* Product usage (collective or average hours)

Distribution method:

* 5% of non-Firefox visitors of https://www.mozilla.org/firefox/new/ (main download page) that match the segment above.

* We will use Optimizely and vary the URL to the download link, set the sample size to, target the segment above, and create two variations for both cohorts. 

* We will probably need to distribute the sample for 7-9 days to get enough data to see any statistical differences in retention over time.

* If there is a chemspill or a dot release of Firefox during the distribution period, the test will need to be stopped. The recommendation is to wait a week after the launch of a major version to reduce the probability of a chemspill after we have created the funnelcakes.

Comment 1

[Chris More [:cmore]]

For the cohort with no suggested tiles, we could flip the browser.newtabpage.enhanced flag to FALSE.

Comment 2

[Olivier Yiptong [:oyiptong]]

That is the correct pref.

This will disable both directory and suggested tiles.

Comment 3

[Kevin Ghim]

Chris, this looks great. Thanks for putting it together. What should we expect the key metrics to track: CTR, time spent, pins/blocks?

Comment 4

[Chris More [:cmore]]

(In reply to Kevin Ghim from comment #3)
> Chris, this looks great. Thanks for putting it together. What should we
> expect the key metrics to track: CTR, time spent, pins/blocks?

For this first round of testing, we're just going to measure retention rate over time of Firefox as a whole. If you already have metrics on tiles (as mentioned in comment 3) that can be filtered by channel (instead of release it will be release-cck-mozilla[the funnelcake id]. So, if the metrics you want to know are already being monitored and you can segment by channel, then yes. If not, we should do this first test and then see what it tells us and then do another more detailed one.

How's that sounds?

Comment 5

[Chris More [:cmore]]

:francesco: can you get all of the logistics setup for a funnelcake test for enhanced tiles? We'll need a funnelcake bug, optimizely bug, mana page, optimizely experiment. It is already added to the opportunity backlog, but just linked to this bug. We'll want to update the link to point to the mana page on the spreadsheet after you have it together.

Comment 6

[Francesco Polizzi [:fpolizzi]]

I can definitely do that. I'll create the bug for the funnelcake and mana doc ASAP so that we can create Optimizely test. Hopefully we can get this built and rolled out by next week.

Comment 7

[Francesco Polizzi [:fpolizzi]]

Building funnelcakes postponed until FX42 release due to a sticky pref bug in the current version.

Reference: https://bugzilla.mozilla.org/show_bug.cgi?id=1208658#c10

Will revisit November 10th.

Comment 8

[Kevin Ghim]

(In reply to Francesco Polizzi [:fpolizzi] from comment #7)
> Building funnelcakes postponed until FX42 release due to a sticky pref bug
> in the current version.
> 
> Reference: https://bugzilla.mozilla.org/show_bug.cgi?id=1208658#c10
> 
> Will revisit November 10th.

Hi Francesco, has the stick pref bug been resolved? I wanted to know if this funnelcake experiment is good to go for Fx42.

Comment 9

[Francesco Polizzi [:fpolizzi]]

(In reply to Kevin Ghim from comment #8)
> (In reply to Francesco Polizzi [:fpolizzi] from comment #7)
> > Building funnelcakes postponed until FX42 release due to a sticky pref bug
> > in the current version.
> > 
> > Reference: https://bugzilla.mozilla.org/show_bug.cgi?id=1208658#c10
> > 
> > Will revisit November 10th.
> 
> Hi Francesco, has the stick pref bug been resolved? I wanted to know if this
> funnelcake experiment is good to go for Fx42.

It appears as though this bug has already been fixed in Fx42 and is riding the trains to full release. We should be good to go for Fx42. :Shane, can you confirm?

Reference: https://bugzilla.mozilla.org/show_bug.cgi?id=1212509

Comment 10

[Chris More [:cmore]]

Heads up. Let's target this funnelcake for Firefox 43 release and not Firefox 42. We already have too many funnelcakes going at the same time and if we don't get enough traffic per funnelcake, the experiment won't be valid.

Let's make these configs for fx43 windows en-US.

Comment 11

[Chris More [:cmore]]

Actually, scratch that, let's target Firefox 44 in January 2016 as the holiday will impact my retention numbers if we target Firefox 43.

Comment 12

[Shane Caraveo (:mixedpuppy)]

The problem did not appear in fx42, but I would suggest additional testing in advance to ensure the targeted version does indeed work correctly.

Comment 13

[Chris More [:cmore]]

We are not going to be running this test in the future. Thanks everyone for all the help.