1304738 - Funnelcake 89/90 for retention experiment

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Winston Bowden]

We're requesting two funnelcake for a test related to retention. 

Goal: Understand if we can impact retention by delivering a message via first run post download
 
OS: Windows

Firefox version: 49.0
 
Language: en-US

Requesting 2 funnel cakes:
We’ll need a funnelcake for the control and a funnel cake for the experiment. The funnelcakes are documented in the funnelcake master tracker: https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0

Funnelcake 89: control
Funnelcake 90: variation 1
 
Time frame:
Requesting funnelcakes asap to be available for QA. We plan to launch the experiment by October 1.

Comment 1

[Chris More [:cmore]]

:nthomas :mkaply: let's talk about this retention test. Separate from onboarding.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Looks like we want the usual ?f=NN suffix on the firstrun page to control content. Keep suppressing the Windows 10 page and extra welcome page like https://github.com/mozilla-partners/funnelcake/blob/master/desktop/funnelcake87/distribution/distribution.ini does ? Do we need to modify anything else, like parameters on search requests ?

Comment 3

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #2)
> Looks like we want the usual ?f=NN suffix on the firstrun page to control
> content. 

Yes, f=89 (control) and f=90 (first run experiment)

> Keep suppressing the Windows 10 page and extra welcome page like
> https://github.com/mozilla-partners/funnelcake/blob/master/desktop/
> funnelcake87/distribution/distribution.ini does ? Do we need to modify
> anything else, like parameters on search requests ?

Yes, no windows 10 or welcome pages. Just the first run. 

The funnelcakes are completely vanilla and just are pref adjustments and the www.mozilla.org is just making a unique /firstrun/?f=(89|90) page that will load based off of these URLs.

:wbowden: can you confirm my replies?

Comment 4

[Winston Bowden]

Confirmed. Can we push these live today?

Comment 5

[Chris More [:cmore]]

:nthomas: what's the estimate on when these funnelcakes could be ready? Need to get this separate experiment started asap. thanks

Comment 6

[Chris More [:cmore]]

This is the URL that should be loaded for funnelcake 90: https://www.mozilla.org/firefox/49.0.1/firstrun/?f=90

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Working on them now.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Configs landed at https://github.com/mozilla-partners/funnelcake/commit/ccc5e4b842ec5e34c9063f5408b20268b9191700

fc89:
http://archive.mozilla.org/pub/firefox/candidates/49.0.1-candidates/build3/funnelcake89/win32/en-US/Firefox%20Setup%2049.0.1.exe (SHA1: 7950c5d5836f92740dab8f95aa461704be6a67f3)

fc90:
http://archive.mozilla.org/pub/firefox/candidates/49.0.1-candidates/build3/funnelcake90/win32/en-US/Firefox%20Setup%2049.0.1.exe (SHA1: 4a108ed33b8a1ef368e328f407b98b7649d101a1)

Fully signed, ready for QA/spot checks.

Bouncer is set up:
 https://download.mozilla.org/?product=firefox-stub-f89&os=win&lang=en-US 
 https://download.mozilla.org/?product=firefox-stub-f90&os=win&lang=en-US 

The builds aren't pushed yet so those links aren't working.

Comment 9

[Eric Renaud [:erenaud]]

Nick - can you please give us an ETA for QA to be completed/when these builds will be pushed?

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

I don't normally cover QA, cmore organizes that for his projects and knows what is involved (setting ni?). I detect some time pressure, but point out we were blocked on the needinfo on Winston.

Comment 11

[Chris More [:cmore]]

Here's what I tested. Embedded below:

I have a busy day today with back-to-back meetings, but I am going to try to squeeze in time to install these buids(In reply to Nick Thomas [:nthomas] from comment #8)
> Configs landed at
> https://github.com/mozilla-partners/funnelcake/commit/
> ccc5e4b842ec5e34c9063f5408b20268b9191700
> 
> fc89:
> http://archive.mozilla.org/pub/firefox/candidates/49.0.1-candidates/build3/
> funnelcake89/win32/en-US/Firefox%20Setup%2049.0.1.exe (SHA1:
> 7950c5d5836f92740dab8f95aa461704be6a67f3)

* no profile to start
* installed fc 89
* open Firefox
* original first run loads with ?f=89 

Status: expected behavior

> 
> fc90:
> http://archive.mozilla.org/pub/firefox/candidates/49.0.1-candidates/build3/
> funnelcake90/win32/en-US/Firefox%20Setup%2049.0.1.exe (SHA1:
> 4a108ed33b8a1ef368e328f407b98b7649d101a1)

* no profile to start
* installed fc 89
* open Firefox
* Experiment first run loads with ?f=90

Status: expected behavior

> 
> Fully signed, ready for QA/spot checks.
> 
> Bouncer is set up:
>  https://download.mozilla.org/?product=firefox-stub-f89&os=win&lang=en-US 
>  https://download.mozilla.org/?product=firefox-stub-f90&os=win&lang=en-US 
> 
> The builds aren't pushed yet so those links aren't working.

These funnelcakes look fine to me since they are just config changes and not actual different binaries. We haven't done simple ones like these for a while. :-)

Comment 12

[Chris More [:cmore]]

nthomas: Green light from me to push to CDN and I will do a full-funnel test from download page to install.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Thanks Chris. I've pushed the builds and the bouncer links are working.

Comment 14

[Chris More [:cmore]]

We're live! Thanks everyone for helping Winston get this out the door. I just started the experiment on /new and it is sharing the US traffic with the v1.2 onboarding experiment. The audiences won't overlap.