Funnelcake 95/96/97 - 50.1.0 retention experiment for advertising sources

RESOLVED FIXED

Status

Release Engineering
Releases: Custom Builds
RESOLVED FIXED
a year ago
a year ago

People

(Reporter: frios, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

a year ago
Hey all, we're requesting funnelcakes to measure effectiveness of advertising campaigns. 

Goal: understand retention of users acquired via Google SEM, and Yahoo display network.

No custom content will be provided, we only need to measure difference in retention based on traffic sources so we'll deliver the funnelcakes through /new using Optimizely.

Here's the master tracker showing funnelcakes 95 (Yahoo ads), 96 (Google SEM ads) and 97 (control): https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0

Date need: 12/20/2016
(Reporter)

Updated

a year ago
Blocks: 1323365
(Reporter)

Updated

a year ago
Summary: Funnelcake 95/96 - retention experiment for advertising sources → Funnelcake 95/96 - 50.0.2 retention experiment for advertising sources
(Assignee)

Comment 1

a year ago
I'll use 50.1.0 which was released a few days ago. We want 32bit windows and just the en-US locale, according to the tracker.
Assignee: nobody → nthomas
Summary: Funnelcake 95/96 - 50.0.2 retention experiment for advertising sources → Funnelcake 95/96/97 - 50.1.0 retention experiment for advertising sources
(Assignee)

Comment 2

a year ago
I'm reviewing the preferences we set in a funnelcake because of bug 1274633, which removed the firstrun and Windows 10 special pages in Firefox 50. Do we need a firstrun page for this test ? eg 
  https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/firstrun/?f=95
Maybe not if it's about longer term retention ? I know we've used hits on firstrun as a check the distribution is working as expected.

Also planning to remove whatsnew page which is shown after updates (startup.homepage_override_url), since that's not shown to the general audience.
Flags: needinfo?(chrismore.bugzilla)
(Assignee)

Updated

a year ago
Flags: needinfo?(frios)
(Reporter)

Comment 3

a year ago
(In reply to Nick Thomas [:nthomas] from comment #2)
> I'm reviewing the preferences we set in a funnelcake because of bug 1274633,
> which removed the firstrun and Windows 10 special pages in Firefox 50. Do we
> need a firstrun page for this test ? eg 
>   https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/firstrun/?f=95
> Maybe not if it's about longer term retention ? I know we've used hits on
> firstrun as a check the distribution is working as expected.
> 
> Also planning to remove whatsnew page which is shown after updates
> (startup.homepage_override_url), since that's not shown to the general
> audience.

Hey Nick, my understanding is that bug 1274633 should only be removing the Windows 10 welcome page and the Learn More page.

https://www.mozilla.org/en-US/firefox/windows-10/welcome/

https://www.mozilla.org/en-US/firefox/46.0.1/firstrun/learnmore/

There's a third page (woah), the First Run page that offers users to sign up for a Firefox Account, should still be available. Also known as the "Almost Done" page.

Is that right^^?
Flags: needinfo?(frios)
(Assignee)

Comment 4

a year ago
Oh right, I got myself confused with the many pages we (used to) open. Vanilla builds do still have the firstrun page via [1] 
 pref("startup.homepage_welcome_url", "https://www.mozilla.org/%LOCALE%/firefox/%VERSION%/firstrun/");

I'll set up the funnelcakes to continue to have a firstrun, just with the ?f=95/96/97 suffix.


[1] https://hg.mozilla.org/releases/mozilla-release/file/FIREFOX_50_1_0_RELEASE/browser/branding/official/pref/firefox-branding.js#l6
Flags: needinfo?(chrismore.bugzilla)
(In reply to Nick Thomas [:nthomas] from comment #5)
> Initial configs landed at
> https://github.com/mozilla-partners/funnelcake/commit/
> ddcd458ad20d8c9e39bf2195269eea6e60bc12d7
> 
> Builds for testing:
> funnelcake95:
> https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> partner-repacks/funnelcake95/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> 5ddd0f7e82205c52716444d30bb46cc9d48baf73)
> 
> funnelcake96:
> https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> partner-repacks/funnelcake96/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> ccf991f8378692f8fd3caf5719643bc4604e4819)
> 
> funnelcake97:
> https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> partner-repacks/funnelcake97/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> 43e89c8b7473dde343888cee2d1ab1cd382d52f8)

Fabio: can you download each of these funnelcakes on a windows machine, kill your profile, install them and confirm:

* distribution.id inside of about:config
* first run URL
Flags: needinfo?(frios)
(Reporter)

Comment 7

a year ago
(In reply to Chris More [:cmore] from comment #6)
> (In reply to Nick Thomas [:nthomas] from comment #5)
> > Initial configs landed at
> > https://github.com/mozilla-partners/funnelcake/commit/
> > ddcd458ad20d8c9e39bf2195269eea6e60bc12d7
> > 
> > Builds for testing:
> > funnelcake95:
> > https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> > partner-repacks/funnelcake95/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> > 5ddd0f7e82205c52716444d30bb46cc9d48baf73)
> > 
> > funnelcake96:
> > https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> > partner-repacks/funnelcake96/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> > ccf991f8378692f8fd3caf5719643bc4604e4819)
> > 
> > funnelcake97:
> > https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> > partner-repacks/funnelcake97/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> > 43e89c8b7473dde343888cee2d1ab1cd382d52f8)
> 
> Fabio: can you download each of these funnelcakes on a windows machine, kill
> your profile, install them and confirm:
> 
> * distribution.id inside of about:config
check
> * first run URL
check

95, 96, 97 all look good. Ready to go.
Flags: needinfo?(frios)
:nthomas: all looks good with these vanilla firefox builds. you're good to push them live and we'll test the link after enabling the test.
Flags: needinfo?(nthomas)
Funnelcakes started on 12/21/2016 at 11am pst.

All looks good. Thanks!
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.