Closed Bug 1323022 Opened 7 years ago Closed 7 years ago

Funnelcake 95/96/97 - 50.1.0 retention experiment for advertising sources

Categories

(Release Engineering :: Release Requests, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: frios, Assigned: nthomas)

References

Details

Attachments

(1 file)

Hey all, we're requesting funnelcakes to measure effectiveness of advertising campaigns. 

Goal: understand retention of users acquired via Google SEM, and Yahoo display network.

No custom content will be provided, we only need to measure difference in retention based on traffic sources so we'll deliver the funnelcakes through /new using Optimizely.

Here's the master tracker showing funnelcakes 95 (Yahoo ads), 96 (Google SEM ads) and 97 (control): https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0

Date need: 12/20/2016
Blocks: 1323365
Summary: Funnelcake 95/96 - retention experiment for advertising sources → Funnelcake 95/96 - 50.0.2 retention experiment for advertising sources
I'll use 50.1.0 which was released a few days ago. We want 32bit windows and just the en-US locale, according to the tracker.
Assignee: nobody → nthomas
Summary: Funnelcake 95/96 - 50.0.2 retention experiment for advertising sources → Funnelcake 95/96/97 - 50.1.0 retention experiment for advertising sources
I'm reviewing the preferences we set in a funnelcake because of bug 1274633, which removed the firstrun and Windows 10 special pages in Firefox 50. Do we need a firstrun page for this test ? eg 
  https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/firstrun/?f=95
Maybe not if it's about longer term retention ? I know we've used hits on firstrun as a check the distribution is working as expected.

Also planning to remove whatsnew page which is shown after updates (startup.homepage_override_url), since that's not shown to the general audience.
Flags: needinfo?(chrismore.bugzilla)
Flags: needinfo?(frios)
(In reply to Nick Thomas [:nthomas] from comment #2)
> I'm reviewing the preferences we set in a funnelcake because of bug 1274633,
> which removed the firstrun and Windows 10 special pages in Firefox 50. Do we
> need a firstrun page for this test ? eg 
>   https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/firstrun/?f=95
> Maybe not if it's about longer term retention ? I know we've used hits on
> firstrun as a check the distribution is working as expected.
> 
> Also planning to remove whatsnew page which is shown after updates
> (startup.homepage_override_url), since that's not shown to the general
> audience.

Hey Nick, my understanding is that bug 1274633 should only be removing the Windows 10 welcome page and the Learn More page.

https://www.mozilla.org/en-US/firefox/windows-10/welcome/

https://www.mozilla.org/en-US/firefox/46.0.1/firstrun/learnmore/

There's a third page (woah), the First Run page that offers users to sign up for a Firefox Account, should still be available. Also known as the "Almost Done" page.

Is that right^^?
Flags: needinfo?(frios)
Oh right, I got myself confused with the many pages we (used to) open. Vanilla builds do still have the firstrun page via [1] 
 pref("startup.homepage_welcome_url", "https://www.mozilla.org/%LOCALE%/firefox/%VERSION%/firstrun/");

I'll set up the funnelcakes to continue to have a firstrun, just with the ?f=95/96/97 suffix.


[1] https://hg.mozilla.org/releases/mozilla-release/file/FIREFOX_50_1_0_RELEASE/browser/branding/official/pref/firefox-branding.js#l6
Flags: needinfo?(chrismore.bugzilla)
(In reply to Nick Thomas [:nthomas] from comment #5)
> Initial configs landed at
> https://github.com/mozilla-partners/funnelcake/commit/
> ddcd458ad20d8c9e39bf2195269eea6e60bc12d7
> 
> Builds for testing:
> funnelcake95:
> https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> partner-repacks/funnelcake95/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> 5ddd0f7e82205c52716444d30bb46cc9d48baf73)
> 
> funnelcake96:
> https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> partner-repacks/funnelcake96/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> ccf991f8378692f8fd3caf5719643bc4604e4819)
> 
> funnelcake97:
> https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> partner-repacks/funnelcake97/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> 43e89c8b7473dde343888cee2d1ab1cd382d52f8)

Fabio: can you download each of these funnelcakes on a windows machine, kill your profile, install them and confirm:

* distribution.id inside of about:config
* first run URL
Flags: needinfo?(frios)
(In reply to Chris More [:cmore] from comment #6)
> (In reply to Nick Thomas [:nthomas] from comment #5)
> > Initial configs landed at
> > https://github.com/mozilla-partners/funnelcake/commit/
> > ddcd458ad20d8c9e39bf2195269eea6e60bc12d7
> > 
> > Builds for testing:
> > funnelcake95:
> > https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> > partner-repacks/funnelcake95/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> > 5ddd0f7e82205c52716444d30bb46cc9d48baf73)
> > 
> > funnelcake96:
> > https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> > partner-repacks/funnelcake96/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> > ccf991f8378692f8fd3caf5719643bc4604e4819)
> > 
> > funnelcake97:
> > https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/
> > partner-repacks/funnelcake97/win32/en-US/Firefox%20Setup%2050.1.0.exe (SHA1:
> > 43e89c8b7473dde343888cee2d1ab1cd382d52f8)
> 
> Fabio: can you download each of these funnelcakes on a windows machine, kill
> your profile, install them and confirm:
> 
> * distribution.id inside of about:config
check
> * first run URL
check

95, 96, 97 all look good. Ready to go.
Flags: needinfo?(frios)
:nthomas: all looks good with these vanilla firefox builds. you're good to push them live and we'll test the link after enabling the test.
Flags: needinfo?(nthomas)
Funnelcakes started on 12/21/2016 at 11am pst.

All looks good. Thanks!
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Component: Custom Release Requests → Release Requests
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: