Closed Bug 1309844 Opened 8 years ago Closed 7 years ago

Funnelcake 105/106 54.0 Win64 A/B testing

Categories

(Release Engineering :: Release Requests, defect, P2)

Product:
Release Engineering
Component:
Release Requests
Platform:
All
Windows
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: RT, Unassigned)

References

Details

User Story

Context: A wider roll-out of 64-bit Firefox on Windows 7/8/8.1/10 is being planned through the stub installer that will make 64-bit Firefox the default build for eligible systems. Ahead of that roll-out we want to test some hypotheses through a bunch of success metrics measured with Telemetry. See the test plan document as a reference for the test.

This bug is for the creation of the test and control funnelcake builds to support the test.

Start date: June 13th
Channel: Release 54
Test plan: https://docs.google.com/document/d/1cmvJUFNr4zg_1_bEXAsbSbEt2RqmN-gKYLrLy2WNnHY/


Audience: non-en-US users, i.e. any language in any country except English users in the United States. (We can’t target en-US users because another team is running a funnelcake targeting English users in the United States at the same time.)
      No description provided.
User Story: (updated)
User Story: (updated)
User Story: (updated)
Blocks: 1309847
cmore, do we need special builds for our Funnelcake test if the only difference between the test and control groups is whether Optimizely tells the /new download page to serve the 32-bit full installer or the 64-bit full installer? Do we need a special build to be able to set users' distribution IDs?
Flags: needinfo?(chrismore.bugzilla)
OS: Unspecified → Windows
Hardware: Unspecified → All
(In reply to Chris Peterson [:cpeterson] from comment #1)
> cmore, do we need special builds for our Funnelcake test if the only
> difference between the test and control groups is whether Optimizely tells
> the /new download page to serve the 32-bit full installer or the 64-bit full
> installer? Do we need a special build to be able to set users' distribution
> IDs?

Well, we do need special funnelcake builds with unique distribution IDs set so that we can segment out those cohorts among the general population. There are two flavors of funnelcakes: simple -- just contain preference changes and complex -- the binary is different or bundled with an add-on. 

I think we can do them with simple ones since the binaries do exist. 

Are we going to be able to do release channel on both cohorts or will we have to mix beta and release together? I mentioned this in the test plan doc.
Flags: needinfo?(chrismore.bugzilla)
(In reply to Chris More [:cmore] from comment #2)
> Are we going to be able to do release channel on both cohorts or will we
> have to mix beta and release together? I mentioned this in the test plan doc.

I added some comments in the test plan. We don't necessarily need to run a beta funnelcake; Romain and I just thought it would be a way to get early data from a Beta 52 funnelcake and then "real" data from a Release 52 funnelcake.

Since we planned to run a Release 52 funnelcake anyways, waiting for it would not delay our 64-bit roll-out schedule.
User story now updated to reflect the fact we target Release 52 to run the tests.
User Story: (updated)
Added these funnelcakes to the master tracker:

https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0

Funnelcake 105: control (win32) f=105
Funnelcake 106: variation (win64) f=106
(In reply to Chris More [:cmore] from comment #5)
> Added these funnelcakes to the master tracker:
> 
> https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-
> 5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0
> 
> Funnelcake 105: control (win32) f=105
> Funnelcake 106: variation (win64) f=106

Romain or cmore, Funnelcake 105 is labeled "control (win32)". Ideally, we would compare 32-bit Firefox running on 64-bit Windows OS (aka "wow64" aka "Windows 32-bit running on Windows 64-bit"). Firefox wow64 users are eligible to run win64 Firefox, but win32 Firefox users (32-bit Firefox running on 32-bit Windows OS) are not eligible and thus not a good control group.

wow64 Firefox can access more memory than win32 but less than wow64. So we would expect wow64 results to be slightly better than win32 and win64 results to be better than wow64.
Flags: needinfo?(rtestard)
Flags: needinfo?(chrismore.bugzilla)
> Romain or cmore, Funnelcake 105 is labeled "control (win32)". Ideally, we
> would compare 32-bit Firefox running on 64-bit Windows OS (aka "wow64" aka
> "Windows 32-bit running on Windows 64-bit"). Firefox wow64 users are
> eligible to run win64 Firefox, but win32 Firefox users (32-bit Firefox
> running on 32-bit Windows OS) are not eligible and thus not a good control
> group.
> 
> wow64 Firefox can access more memory than win32 but less than wow64. So we
> would expect wow64 results to be slightly better than win32 and win64
> results to be better than wow64.

Agreed, we should only focus on 64 bit OS users. I although don't recall how - can we detect 64 bit Win7+ users from the /new page or do we have to filter 64 bit OS users in Telemetry?
Flags: needinfo?(rtestard)
(In reply to Romain Testard [:RT] from comment #7)
> Agreed, we should only focus on 64 bit OS users. I although don't recall how
> - can we detect 64 bit Win7+ users from the /new page or do we have to
> filter 64 bit OS users in Telemetry?

The /new page can identify 64-bit OS users by their User-Agent string, regardless of the browser they are using to download Firefox. 64-bit OS users' User-Agent string will contain the substring "Win64" or "WOW64".
Thanks, I added details in the user story field of bug 1309847 so it's clear we want to target Win7+ users on 64 bit OS for test and control cohorts.
(In reply to Chris Peterson [:cpeterson] from comment #6)
> (In reply to Chris More [:cmore] from comment #5)
> > Added these funnelcakes to the master tracker:
> > 
> > https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-
> > 5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0
> > 
> > Funnelcake 105: control (win32) f=105
> > Funnelcake 106: variation (win64) f=106
> 
> Romain or cmore, Funnelcake 105 is labeled "control (win32)". Ideally, we
> would compare 32-bit Firefox running on 64-bit Windows OS (aka "wow64" aka
> "Windows 32-bit running on Windows 64-bit"). Firefox wow64 users are
> eligible to run win64 Firefox, but win32 Firefox users (32-bit Firefox
> running on 32-bit Windows OS) are not eligible and thus not a good control
> group.
> 
> wow64 Firefox can access more memory than win32 but less than wow64. So we
> would expect wow64 results to be slightly better than win32 and win64
> results to be better than wow64.

Yeah, I meant that too and made it more explicit in the funnelcake master track (https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0) and thanks :rt for making a note in that bug. So, both will target Win64 OSs, but one will get win32 build and the other will get win64.
Flags: needinfo?(chrismore.bugzilla)
We're having to delay this by 2 releases because of a flash bug (bug 1336517). A work-around is being implemented with FF54 (bug 1340276) so FF54 is the sooner we can run the funnelcake.
User story field now updated.
User Story: (updated)
Priority: -- → P2
(In reply to Romain Testard [:RT] from comment #11)
> We're having to delay this by 2 releases because of a flash bug (bug
> 1336517). A work-around is being implemented with FF54 (bug 1340276) so FF54
> is the sooner we can run the funnelcake.
> User story field now updated.

:RT do you want to keep the same funnelcake 105 and 106 for this experiment for use at a later date?
(In reply to Chris More [:cmore] from comment #12)
> (In reply to Romain Testard [:RT] from comment #11)
> > We're having to delay this by 2 releases because of a flash bug (bug
> > 1336517). A work-around is being implemented with FF54 (bug 1340276) so FF54
> > is the sooner we can run the funnelcake.
> > User story field now updated.
> 
> :RT do you want to keep the same funnelcake 105 and 106 for this experiment
> for use at a later date?

Yes, that works. FYI the async drawing bug test is going well so we're still targeting FF54 for funnelcake.
Chris, is comment #14 now a firm requirement ?
Flags: needinfo?(cpeterson)
Summary: Funnelcake build request to support Win64 A/B testing → Funnelcake 105/106 54.0 Win64 A/B testing
Bug 1340934 makes me think not. 

A separate queation, is this going to be all locales or just en-US ?
en-US only
(In reply to Nick Thomas [:nthomas] from comment #16)
> Bug 1340934 makes me think not. 
My understanding is that changing preferences is not needed anymore:
- Flash async drawing was re-enabled on Beta 54 64 bit builds per https://bugzilla.mozilla.org/show_bug.cgi?id=1363876
- Flash async drawing is kept disabled on 32 bit builds.
(In reply to Nick Thomas [:nthomas] from comment #15)
> Chris, is comment #14 now a firm requirement ?

No. As Romain says in comment 18, we don't need to set any special prefs for the Funnelcake builds (because Flash async drawing is enabled by default for 64-bit Firefox).

I will mark my comment 14 as obsolete.
Flags: needinfo?(cpeterson)
Thanks for the clarifications. Funnelcake configs pushed with https://github.com/mozilla-partners/funnelcake/commit/92f543a9ca741a9252f9e2a28b271c58c6b5131f. Mihai and/or Aki will generate builds once 54.0 RC1 gets under way.
Do we have any details about when we want to start this funnelcake (ie w.r.t to potential point releases), and how long we'll be distributing it to get adequate cohort sizes ?
(In reply to Nick Thomas [:nthomas] from comment #21)
> Do we have any details about when we want to start this funnelcake (ie w.r.t
> to potential point releases), and how long we'll be distributing it to get
> adequate cohort sizes ?

AFAIK, there is no need to wait for any 54 point releases before distributing the Funnelcake.

@ cmore, how long do you recommend that we distribute the 64-bit Funnelcake? I think Romain said we would distribute the Funnelcake for either 1 or 2 weeks (and then run the experiment for 4 weeks).
Flags: needinfo?(chrismore.bugzilla)
(In reply to Chris Peterson [:cpeterson] from comment #22)
> (In reply to Nick Thomas [:nthomas] from comment #21)
> > Do we have any details about when we want to start this funnelcake (ie w.r.t
> > to potential point releases), and how long we'll be distributing it to get
> > adequate cohort sizes ?
> 
> AFAIK, there is no need to wait for any 54 point releases before
> distributing the Funnelcake.
> 
> @ cmore, how long do you recommend that we distribute the 64-bit Funnelcake?
> I think Romain said we would distribute the Funnelcake for either 1 or 2
> weeks (and then run the experiment for 4 weeks).

I would go for 60k downloads for this specific build for each funnelcake for a total of 120k downloads. 

What is average amount of en-US win64 downloads we get per week?
Flags: needinfo?(chrismore.bugzilla) → needinfo?(cpeterson)
(In reply to Chris More [:cmore] from comment #23)
> What is average amount of en-US win64 downloads we get per week?

From my calculations, we get about 2.7M downloads per week from users running 64-bit Windows OS. They still get 32-bit Firefox builds by default, but would be eligible for 64-bit Firefox.
Flags: needinfo?(cpeterson)
(In reply to Nick Thomas [:nthomas] from comment #20)
> Thanks for the clarifications. Funnelcake configs pushed with
> https://github.com/mozilla-partners/funnelcake/commit/
> 92f543a9ca741a9252f9e2a28b271c58c6b5131f. Mihai and/or Aki will generate
> builds once 54.0 RC1 gets under way.

Nick, do you need any more information to deploy this Funnelcake ? 54.0 RC2 builds are now available.

I believe the plan is:

- 60k downloads for each Funnelcake build (experiment and control) for a total of 120k downloads.
- distributed over 1 or 2 weeks? (and then run the experiment for 4 weeks) Is Funnelcake distribution is throttled to avoid selecting all 120k testers from the same day of the week?
Flags: needinfo?(nthomas)
Thanks for figuring out duration. FWIW, it's not up to Releng to implement the www.mozilla.org changes, just wanted to make sure we had thought about that as a larger group. W.r.t to when we start, earlier gets you data sooner but we're almost certain to have to update the funnelcake when a point release comes along. It's not a huge deal, just need to know for planning porpoises.

Aki or Mihai, would you mind generating the funnelcakes based on 54.0 build3 ? The config is ready and landed.
Flags: needinfo?(nthomas)
Chris, can you please confirm all is set on the Optimizely side. Bug 1309847 tracks the Optimizely set-up, please let me know if I should contact someone else for this. Also is any testing needed before going live?
http://archive.mozilla.org/pub/firefox/candidates/54.0-candidates/build3/partner-repacks/funnelcake105/v1/win32/en-US/Firefox%20Setup%2054.0.exe
SHA256(repacked_builds/signed/partner-repacks/funnelcake105/v1/win32/en-US/Firefox Setup 54.0.exe)= 9ea08e7e624e75bd242f1ecc940b5ccf97dd86784c333e4be8122e4dbd56b279

http://archive.mozilla.org/pub/firefox/candidates/54.0-candidates/build3/partner-repacks/funnelcake106/v1/win64/en-US/Firefox%20Setup%2054.0.exe
SHA256(repacked_builds/signed/partner-repacks/funnelcake106/v1/win64/en-US/Firefox Setup 54.0.exe)= fef226d5edd1cdb39ebfb7d1043c0c302adccf72b510e5ee615e84b7227c4365
(In reply to Romain Testard [:RT] from comment #27)
> Chris, can you please confirm all is set on the Optimizely side. Bug 1309847
> tracks the Optimizely set-up, please let me know if I should contact someone
> else for this. Also is any testing needed before going live?

cmore ^
Flags: needinfo?(chrismore.bugzilla)
(In reply to Chris Peterson [:cpeterson] from comment #29)
> (In reply to Romain Testard [:RT] from comment #27)
> > Chris, can you please confirm all is set on the Optimizely side. Bug 1309847
> > tracks the Optimizely set-up, please let me know if I should contact someone
> > else for this. Also is any testing needed before going live?
> 
> cmore ^

We recently changed to try not use Optimizely anymore for these type of redirect experiments as the mozilla.org team has built a tool called Traffic Cop that works fine for these distributions. Bug 1309847 should be a general "Create experiment to distribute Win 64 funnelcake experiment".

Since the mozilla.org team wants to use Traffic Cop going forward, they need to set up bug 1309847 as a Traffic Cop experiment and do the coding to make that possible. Eric Renaud is the program manager for the team and he should take that bug and get their devs to set it up.

I don't set these up anymore as the mozilla.org has expressed that they want to manage the larger experiments on their site and use Traffic Cop as much as possible. With that, let's needinfo Eric Rendaud and get bug 1309847 cleared up to be Traffic Cop and have them do the pull request needed to perform the experiment.
Flags: needinfo?(chrismore.bugzilla) → needinfo?(erenaud)
I've asked on bug 1309847 if we're doing bouncer links the same way as we used to with Optimizely. None of that is set up, we can't ship this funnelcake yet.
Eric, can you please clarify when this could be done by? This funnelcake test timing is quite critical since the results from the experiment will gate shipping 64 bit on release for new usrers with FF55.
User Story: (updated)
We're working on this in tandem with the search retention funnelcake.  It'll be the same pull request (considering the 95/5 percent allocations and the potential concurrent 'hairiness' there) for both. 

That (search retention) is slated for 6/20, so that's when both tests will kick off. Jon Petto is handling the whole of it. He's aware of the latest update to non en-US as well.

As cmore has pointed out, please needinfo for any future funnelcake experiments (preferably when the bug is first submitted so I can forecast for resourcing needs). Thanks!
Flags: needinfo?(erenaud) → needinfo?(rtestard)
Per bug 1309847 comment #11 and #14 I've updated the locale list to "de es-ES fr pt-BR ru", as well as the upload location, in https://github.com/mozilla-partners/funnelcake/commit/4b8746a6236a86e08144a5401255d7fd359f737b. Spun up new builds:

funnelcake105:
https://archive.mozilla.org/pub/firefox/candidates/54.0-candidates/build3/partner-repacks/funnelcake105/v2/win32/

SHA256(funnelcake105/v2/win32/de/Firefox Setup 54.0.exe)= 1650d5f63b211e1aaf6172b95f855735fabfd0cf4a278aac371adcff89e83155
SHA256(funnelcake105/v2/win32/es-ES/Firefox Setup 54.0.exe)= fa8a8e9e1ce90f870fc15daed9d344db7dd0d306d1dc30c41efd724901319c6c
SHA256(funnelcake105/v2/win32/fr/Firefox Setup 54.0.exe)= 50dc9e7cf91aa89ec27ea772d45aec2025dd7ec6a2ee5e2f9fb05dd26314f29c
SHA256(funnelcake105/v2/win32/pt-BR/Firefox Setup 54.0.exe)= df9ec8a839d717a5983ff973fec1e8b31a5e506bfbe175cbe1d6ff24776e687d
SHA256(funnelcake105/v2/win32/ru/Firefox Setup 54.0.exe)= b9916967d9c2530d646c0bddbe11b4de8e721383d6c41ad05d19d8768f4f12ea

funnelcake106:
https://archive.mozilla.org/pub/firefox/candidates/54.0-candidates/build3/partner-repacks/funnelcake106/v2/win64/

SHA256(funnelcake106/v2/win64/de/Firefox Setup 54.0.exe)= 507822de55dbb33feba80ad8c332d6edc96969070000d3ef9118a76bcaedf478
SHA256(funnelcake106/v2/win64/es-ES/Firefox Setup 54.0.exe)= 485f9ffed48078f4d04a4045c18a600ce7f1107d55351ed0aab95c06713e6d08
SHA256(funnelcake106/v2/win64/fr/Firefox Setup 54.0.exe)= 10813fbb41ea2b17f3f9cc2474f38f99775232803f4fbfd0c094011c5c78be76
SHA256(funnelcake106/v2/win64/pt-BR/Firefox Setup 54.0.exe)= 54c7fa6f15b096a3b66c6c794a889d6208076da48dac4bf1def183563e35792f
SHA256(funnelcake106/v2/win64/ru/Firefox Setup 54.0.exe)= d4bd5f1be46ace1fe14ecdb55b777e08c49c6bbd12489c0e1b17b1c938e54e99


Also setup bouncer *for testing*, eg
https://download.mozilla.org/?product=firefox-stub-f105&os=win&lang=de
https://download.mozilla.org/?product=firefox-stub-f106&os=win64&lang=fr

Substitute other locale codes as needed. I needed to add some 0-byte, fake installers for zh-TW so that download.m.o would serve the redirects.
Chris, have you thought about engaging QA here ? The last few funnelcakes we've asked Grover Wimberly from SoftVision US to help test end to end. So far I've just smoke tested a couple of builds to make sure the firstrun page has the ?f=105 or 106 query parameter, and the update channel is set correctly.

Releng TODO: respond to any respin requests, move files to final locations in firefox/releases/, update bouncer links; respin as needed for 54.0.x point releases until distribution complete.
Flags: needinfo?(cpeterson)
(In reply to Nick Thomas [:nthomas] from comment #34)
> Per bug 1309847 comment #11 and #14 I've updated the locale list to "de
> es-ES fr pt-BR ru", as well as the upload location, in
> https://github.com/mozilla-partners/funnelcake/commit/
> 4b8746a6236a86e08144a5401255d7fd359f737b. Spun up new builds:

Thanks!

(In reply to Nick Thomas [:nthomas] from comment #35)
> Chris, have you thought about engaging QA here ? The last few funnelcakes
> we've asked Grover Wimberly from SoftVision US to help test end to end. So
> far I've just smoke tested a couple of builds to make sure the firstrun page
> has the ?f=105 or 106 query parameter, and the update channel is set
> correctly.

That's a good idea. I'll follow up with Romain and Softvision.
Flags: needinfo?(cpeterson)
User Story: (updated)
Flags: needinfo?(rtestard)
(In reply to Romain Testard [:RT] from comment #37)
> Grover, I'm following-up from my separate e-mail.
> Given previous testing of the streamlined stub, defaulting to Win64 and 2Gb
> threshold it sounds like what needs attention is making sure we deliver the
> right funnelcake to the right cohort but please let me know what you think.
> 
> - The streamlined stub installer is used, the 2Gb memory requirement is
> applied
> - Only users of 64 bit Win7+ OS with over 2Gb get test funnelcake 106
> - Only users of 64 bit Win7+ OS get test funnelcake 105

I got that wrong, please ignore my previous comment, we use full installers are initially agreed, what needs attention is:
- The full installers are used, no memory requirement is applied
- Only users of 64 bit Win7+ OS get test funnelcake 106 "de es-ES fr pt-BR ru"
- Only users of 64 bit Win7+ OS get test funnelcake 105 "de es-ES fr pt-BR ru"
We can check the builds are OK, but I don't know if we have a staging system for www.mozilla.org + TrafficCop to simulate handing them out. Bug 1309847 would be a good place to ask.
Hi all, catching/clarifying up on comments on the bug here as I write up some test cases just to make sure I'm on the same page. The builds in Comment 34 are what we'll be using to test/verify?
Flags: needinfo?(rtestard)
Flags: needinfo?(nthomas)
Flags: needinfo?(cpeterson)
jpetto has a question in the Traffic Cop experiment bug that I want to make sure gets answered asap: https://bugzilla.mozilla.org/show_bug.cgi?id=1309847#c20
(In reply to Grover Wimberly IV [:Grover-QA] from comment #41)
> Comment 34 are what we'll be using to test/verify?

Yes. I suggest using the bouncer links at the end of the comment.
Flags: needinfo?(nthomas)
QA reports seem fine, so I pushed the files from the candidates to the releases directory, eg https://archive.mozilla.org/pub/firefox/releases/54.0-funnelcake105/

Also updated the bouncer links to point there, and verified they're working. That's it for RelEng, over to the website people to make this live.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
jpetto says he will be pushing the funnelcake to mozilla.org prod in about an hour. I tested his fixes for the bedrock/bouncer URL bugs reported by Softvision in bug 1309847 and they worked for me. We can retest on prod soon.
Flags: needinfo?(rtestard)
Flags: needinfo?(cpeterson)
Component: Custom Release Requests → Release Requests
You need to log in before you can comment on or make changes to this bug.