Closed Bug 1191060 Opened 9 years ago Closed 9 years ago

Funnelcake build to test impact of default browser prompt

Categories

(Release Engineering :: Release Requests, defect)

Product:
Release Engineering
Component:
Release Requests
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Dolske, Assigned: coop)

References

(Blocks 1 open bug)

Details

(Whiteboard: [fxgrowth] funnelcake60 funnelcake61)

Attachments

(1 file, 1 obsolete file)

Patch v.2
2.04 KB, patch
nthomas
: review+
Details | Diff | Splinter Review
We'd like to do a funnelcake to test the impact of the default browser setting, by having a funnelcake build that does _not_ set Firefox as the default browser. We can then use this as a worst-case baseline (in addition to normal installs on <Win8 as an ideal baseline), to help guide possible future improvements to this prompting. The in-product change here is simple -- the Funnelcake build can just flip browser.shell.checkDefaultBrowser to false. We *could* additionally have a funnelcake stub installer with the opt-out default browser setting disabled. But after thinking about this, I don't think it's worth doing... It's already disabled on Windows 8/10 (where apps can't change defaults without OS intervention), and a significant portion of downloads (~1/4) come from 8+10. It might be academically interesting to see how important this is to Windows 7/XP users, but there's little reason to remove this feature from the stub even if we found it has no impact. Also, leaving it enabled on Win7/XP may let us concurrently see the impact of the prompting over the long term (ie, a set of users where Firefox was set as the default at install-time, but we never prompted again if it got changed). Looks like bug 1184279 and bug 1157556 are recent examples of how to make funnelcake build changes. Also, note that in bug 951627 / bug 1086958 we tried switching to a notification bar for Firefox 34, but backed it out prior to release because FHR apparently showed a drop. I'm not familiar with that data or how significant it was, but would be interesting to find it and compare against what we find in this funnelcake.
(In reply to Justin Dolske [:Dolske] from comment #0) > The in-product change here is simple -- the Funnelcake build can just flip > browser.shell.checkDefaultBrowser to false. Am I correct in assuming that with an approach like this we could just do a repack build of current production builds? If so, I like this. :)
Funnelcakes are just a distribution repack. You can put the funnelcake together yourself, fork https://github.com/mozilla-partners/funnelcake and increment the number, make the changes, do a pull request. However there is a spreadsheet somewhere to track the funnelcake build numbers, etc. cmore would know more.
Flags: needinfo?(chrismore.bugzilla)
Should we target 40.0 for this, given it ships in about a week, and possibly wait a little longer to avoid any 40.0.1. Do you have any opinions on that cmore ? How long would be need to have downloads enabled to get a representative sample size ? Perhaps there's still time to do it with 39.0. dolske, sounds like this is windows only, but what set of locales ? We often use a control funnelcake to measure against, will that be useful here too ? Please note that we need to remove the pref browser.startup.homepage at https://github.com/mozilla-partners/funnelcake/blob/master/desktop/funnelcake41/distribution/distribution.ini#L13, and you may not want a firstrun page either.
I would rather target Firefox 40.0 or later as I don't like to be distributing an older version of Firefox even though it would still update over time. How about we wait to 40.0 is out the door, if there is no immediate chemspill, we make a funnelcake 40.0 for this test, and I will distribute it via Optimizely on the /firefox/new/.
Flags: needinfo?(chrismore.bugzilla)
This is the funnelcake master tracker: https://docs.google.com/a/mozilla.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit?usp=sharing The next available funnelcake ID is 42 and since we need to do an A/B test, we'll need two funnelcakes, #42 and #43. Will the build be en-US windows?
(In reply to Chris More [:cmore] from comment #5) > This is the funnelcake master tracker: > > https://docs.google.com/a/mozilla.com/spreadsheets/d/1gZ-GFyubAAlIxh-u- > 5SR52rqKKfBRLRuwrjSq97QkWA/edit?usp=sharing > > The next available funnelcake ID is 42 and since we need to do an A/B test, > we'll need two funnelcakes, #42 and #43. I'll reserve those numbers now! :) > Will the build be en-US windows? Not sure, I don't think we need to restrict ourselves to en-US. Definitely Windows only.
This and bug 1191472 need to duke it out over sequencing.
I assume that we are doing this experiment across all platforms. In that case there's one pretty important case we need to consider: Windows XP, Vista and 7 come with a checkbox in the installer that sets the default browser, meaning that the prompt in the product will almost never show up on these OSes. If we want to know what impact being the default has, we need to deactivate that too.
Sheila: what's the status of this funnelcake request? I think it would be best that I create a test details page on this funnelcake test to ensure we are all on the same page on what this means.
Flags: needinfo?(smooney)
Is everyone cool if I create a test details page for this? It will be something like this: https://mana.mozilla.org/wiki/display/FIREFOX/CTA+to+Download+Firefox+Android+on+Scene+2+of+Desktop+Download+-+Take+2
(In reply to Chris More [:cmore] from comment #10) > Is everyone cool if I create a test details page for this? It will be > something like this: > https://mana.mozilla.org/wiki/display/FIREFOX/ > CTA+to+Download+Firefox+Android+on+Scene+2+of+Desktop+Download+-+Take+2 Is it necessary that the funnelcake details page be on Mana? We should keep this publicly visible unless there is a compelling reason not to.
(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #11) > (In reply to Chris More [:cmore] from comment #10) > > Is everyone cool if I create a test details page for this? It will be > > something like this: > > https://mana.mozilla.org/wiki/display/FIREFOX/ > > CTA+to+Download+Firefox+Android+on+Scene+2+of+Desktop+Download+-+Take+2 > > Is it necessary that the funnelcake details page be on Mana? We should keep > this publicly visible unless there is a compelling reason not to. Yes, we can make this public. I'll get it somewhere public after I have it together.
Sorry for the slow response here... I concur(red) with previous comments about this not being on fire, and something that can happen sometime after the release of Firefox 40 and any chemspills (which should be wrapping up now, so we can think about moving this back to the active burner.) (In reply to Philipp Sackl [:phlsa] (Firefox UX) please use needinfo from comment #8) > I assume that we are doing this experiment across all platforms. > In that case there's one pretty important case we need to consider: Windows > XP, Vista and 7 come with a checkbox in the installer that sets the default > browser, meaning that the prompt in the product will almost never show up on > these OSes. Oh, right, good point. So... I think this would require a (trivially) customized stub installer. Looks like we build the installer for funnelcakes (since it knows to download the funnelcake build from a different URL), so I presume this shouldn't be a big deal? (In reply to Chris More [:cmore] from comment #12) > Yes, we can make this public. I'll get it somewhere public after I have it > together. WFM.
Flags: needinfo?(nthomas)
Stub installers are actually a higher bar to cross, because they require a compilation to use a different url for the full installer. Or some Apache hackery on download.m.o to redirect some proportion of requests from the normal stub to funnelcake full installers.
Flags: needinfo?(nthomas)
Oh, we normally use full installers for the funnelcakes... But looks like that also sets the default browser, so we'd still need a customized installer to avoid it. Actually, I already considered this in comment 0 (ancient history -- 9 days ago!), and I think we just don't need to worry about what the installer does.
Most funnelcakes we have done in the past use the full installer. So, we would need need two funnelcakes. Funnelcake 42: no default prompt Funnelcake 43: control with as-is prompt Build: win32 full installer (no stub) Language: en-US Version: Since we are half way through the 40.0 release, we should target the 41.0 release. We'll distribute both full installers as a sampled A/B test to Windows 10 non-Firefox visitors to the https://www.mozilla.org/firefox/new/ page for at least a week. Probably at 10% sample should be enough traffic, but we can monitor. Typically, for an A/B test with funnelcakes, I try to get about 25k installs per variation. Is everyone cool with doing this for 41.0, or is there a push to try to squeeze it in before 41.0 is here? We need typically 7 days to get a good sample size and that means we would need to get this out the door no later than September 14th if we want to do it before 41.0.
Let's not break the funnelcake numbering here. That the IDs match the release version is super useful, we should preserve that. Instead, we should only need two IDs: mozilla41 (normal funnelcake build) and mozilla41-ndp (no default prompt), since the control group can also be the normal funnelcake for that release.
(In reply to Mike Connor [:mconnor] from comment #17) > Let's not break the funnelcake numbering here. That the IDs match the > release version is super useful, we should preserve that. Instead, we > should only need two IDs: mozilla41 (normal funnelcake build) and > mozilla41-ndp (no default prompt), since the control group can also be the > normal funnelcake for that release. mconnor: Of the 20+ funnelcakes I've done in the past few years, we've never once purposely aligned the funnelcake ID to the Firefox version on purpose. Nthomas and I have been just treating the funnelcake ID as an incrementing number. We typically create a separate funnelcake for each variation with its each own unique number. A funnelcake we are running now is #44. See: https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0
Flags: needinfo?(mconnor)
Follow up hypothesis from mmayo after we get an answer to this test: Users who are prompted for default status after they have used Firefox a few times (3-5 times), will more likely set Firefox as default.
Blocks: 1198541
Whiteboard: [fxgrowth]
Attached patch Patch v.1 (obsolete) — — Splinter Review
Ok, this is patch against https://hg.mozilla.org/build/partner-repacks, cribbing off the funnelcake44 that was there. This adds a funnelcake42 (which flips browser.shell.checkDefaultBrowser and browser.usedOnWindows10), and a funnelcake43 (the null/controll group). I wasn't sure about the existing startup.homepage_welcome_url / startup.homepage_override_url in distribution.ini... I think these are the same as stock Firefox, just with an extra URL param to identify the funnelcake version. Are these useful for understanding Funnelcake usage, or is this only really interesting if one was actually A/B testing those pages? [In other words: should I keep these in distribution.ini or remove them?] Right now those pages just seem to promote Pocket or FxA signups, so they shouldn't interfere with measuring default-prompting impact. (Obviously if these were pushing "Set Firefox as your default!" that would be different.) The only other thought that comes to mind is there are some about:home snippets running right now regarding Windows 10. I'm assuming that we shouldn't worry about that, between (A) general low engagement/impact of them and (2) that being somewhat separate from what we're specifically testing here (ie, the browser explicitly prompting you to set it as a default).
Assignee: nobody → dolske
Attachment #8655075 - Flags: feedback?(nthomas)
Comment on attachment 8655075 [details] [diff] [review] Patch v.1 >diff --git a/inactive-configs/funnelcake42/distribution/distribution.ini b/inactive-configs/funnelcake42/distribution/distribution.ini >new file mode 100644 >--- /dev/null >+++ b/inactive-configs/funnelcake42/distribution/distribution.ini Please use partners/ instead of inactive-configs/ for all of these. Otherwise this looks fine. (In reply to Justin Dolske [:Dolske] from comment #20) > I wasn't sure about the existing startup.homepage_welcome_url / > startup.homepage_override_url in distribution.ini... I think these are the > same as stock Firefox, just with an extra URL param to identify the > funnelcake version. Are these useful for understanding Funnelcake usage, or > is this only really interesting if one was actually A/B testing those pages? > [In other words: should I keep these in distribution.ini or remove them?] In the past they've been used for studying the funnel, eg # of downloads vs # of installed-and-run-once (via firstrun) vs long term usage. Maybe it's better to just leave them as is, so the first-run UX is consistent with other downloads, and you get a rough counter of sample size. The whatsnew is being enabled/disabled in the update server, and in fact overriding anything set in the app or distribution.
Attachment #8655075 - Flags: feedback?(nthomas) → feedback+
Attached patch Patch v.2 — — Splinter Review
Moved to partners/. What are the next steps here? I assume releng just needs to kick off the repacks, and the cmore can feed it into Optimizely?
Attachment #8655075 - Attachment is obsolete: true
Attachment #8658365 - Flags: review?(nthomas)
(In reply to Justin Dolske [:Dolske] from comment #22) > Created attachment 8658365 [details] [diff] [review] > Patch v.2 > > Moved to partners/. > > What are the next steps here? I assume releng just needs to kick off the > repacks, and the cmore can feed it into Optimizely? Yeah, I just need the builds available to QA made from 41.0 en-US win32 builds. After we QA, then we can get them up on download.mozilla.org. Then I do the test with optimizely. The config look like and match the specs in comment 16.
(In reply to Chris More [:cmore] from comment #16) > Is everyone cool with doing this for 41.0, or is there a push to try to > squeeze it in before 41.0 is here? We need typically 7 days to get a good > sample size and that means we would need to get this out the door no later > than September 14th if we want to do it before 41.0. Oops, I missed this. I don't think it's the end of the world if we do it for 41, but if we can get this done for 40 that would be ideal. (Mostly just because we know it will take a while to have a data to look at, and default browser prompting is an area of active interest.) Is this still doable if Nick is able to get builds done quickly?
Set up bouncer, urls will be https://download.mozilla.org/?product=firefox-40.0.3-SSL-f42&os=win&lang=en-US https://download.mozilla.org/?product=firefox-40.0.3-SSL-f43&os=win&lang=en-US Push command for RelEng: # ffxbld@stage rsync -av /pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1191060/funnelcake42/ /pub/mozilla.org/firefox/releases/40.0.3-funnelcake42/ rsync -av /pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1191060/funnelcake43/ /pub/mozilla.org/firefox/releases/40.0.3-funnelcake43/ Also added the config to the new github system for posterity.
(In reply to Nick Thomas [:nthomas] from comment #26) > dolske, the builds are in > https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3- > candidates/build1/partner-repacks-bug1191060/ > for verification. I confirmed that installing these works as expected on my Windows 10 desktop. For each, I deleted my existing profiles, installed, and checked what happened on launch... The F42 build correctly skips showing the default browser prompt and the first-time-on-win10 page. The F43 (control) shows both.
cmore, are you OK to go live then ?
Flags: needinfo?(chrismore.bugzilla)
NB: The bouncer links don't work yet, I'm just making sure that we're good to go per comment #23 and 24.
Let's do this. We have less than 2 weeks until Firefox 41.0 and thus we are on the edge of getting enough data, but I will bump up the sample once I see data coming in. Please get the funnelcake builds live on D.M.O and I will get the optimizely test set up.
Flags: needinfo?(chrismore.bugzilla)
Bouncer links are live.
Builds are done and published.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Flags: needinfo?(smooney)
Flags: needinfo?(mconnor)
:dolske: can I run for this for all versions of Windows or do we need it only for Windows 10? Just need to confirm.
Yes, this can run for all versions.
Do we care to specifically know Windows 7 vs Windows 8 vs Windows 10 to see if there are any differences? I looked in about:healthreport on Windows 10 machine, and I don't see the osVersion key value pair in the raw data. I will look again. I just want to make sure we can segment the data to be able to see it by OS version.
We do need to filter, because the installer still sets Firefox as the default on < Win 8 (see comment 0). So we really only expect the funnelcake to work differently on Win8/Win10, where the installer can't do this and we rely on prompting the user to go through the OS default-selection UI. I don't really understand FHR, but surely it collects the OS version somewhere? Vladan?
Flags: needinfo?(vladan.bugzilla)
I can make the test be Windows 8 + Windows 10. I just want to make sure we can see windows 8 & 10 in FHR data.
I can use this JS check to see if the user is on Windows 8 or Windows 10: navigator.userAgent.indexOf('Windows NT 6\.2') + navigator.userAgent.indexOf('Windows NT 10\.0') > -1;
(In reply to Justin Dolske [:Dolske] from comment #37) > We do need to filter, because the installer still sets Firefox as the > default on < Win 8 (see comment 0). So we really only expect the funnelcake > to work differently on Win8/Win10, where the installer can't do this and we > rely on prompting the user to go through the OS default-selection UI. > > I don't really understand FHR, but surely it collects the OS version > somewhere? Vladan? I found it in FHR. It changed locations from the definitions page: org.mozilla.sysinfo.version and the value is 10.0 in Windows 10.
Funnelcake test has been started.
(In reply to Justin Dolske [:Dolske] from comment #37) > I don't really understand FHR, but surely it collects the OS version > somewhere? Vladan? I don't know FHR, I only know Telemetry, but I did look in about:healthreport and I saw that there is a "org.mozilla.sysinfo.sysinfo" probe and it that it has a "version" field which report the OS version: "org.mozilla.sysinfo.sysinfo": { "_v": 2, "cpuCount": 8, "isWow64": 1, "memoryMB": 8079, "architecture": "x86", "name": "Windows_NT", "version": "6.1" }
Flags: needinfo?(vladan.bugzilla)
What Chris said :)
+sguha as he has been conducting fairly extensive investigations of the impact of Win10 on default status using FHR v2 and v4.
Heads up. We have to re-run this funnelcake, thus we need two more funnelcakes with the same config. We'll probably have to use new funnelcake numbers and just update the old configs. The reason we have to re-run the test is because of the loss of FHR records (unrelated to test) during period of when we were measuring retention.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Just created the PR for these configs. Once those are reviewed and merged I will NI Nick for a rebuild. https://github.com/mozilla-partners/funnelcake/pull/8
coop's handling funnelcake builds at the moment, cheers.
:coop: check out (and review) comment 46 for francesco's config. These can go out after the fall campaign funnelcake in bug 1216740. We can probably launch these the second week in November -- pending any chemspills.
Flags: needinfo?(coop)
(In reply to Chris More [:cmore] from comment #48) > :coop: check out (and review) comment 46 for francesco's config. These can > go out after the fall campaign funnelcake in bug 1216740. We can probably > launch these the second week in November -- pending any chemspills. OK, I'll queue this up for late next week.
Assignee: dolske → coop
Flags: needinfo?(coop)
Whiteboard: [fxgrowth] → [fxgrowth] funnelcake60 funnelcake61
Optimizely test updated and ready for QA. Things changed: - Scheduler set to end Dec. 15 at 11:00 PM GMT -8. - URL's for both variations. https://app.optimizely.com/edit?experiment_id=3449261060
Flags: needinfo?(chrismore.bugzilla)
Funnelcake test re-launched. Thanks everyone
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Flags: needinfo?(chrismore.bugzilla)
Resolution: --- → FIXED
Component: Custom Release Requests → Release Requests
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: