1332726 - Funnelcakes 101/102/103/104 51.0 for Onboarding v1.6 Add-on Experiment (3/2/2017 launch)

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect, P2)

Description

[Ben Niolet]

We are building a new version of the onboarding add-on experiment. (https://github.com/mozilla/all-aboard)

The experiment will launch with a version of Firefox bundled with the Version 1.6 Add-on XPI. Our target launch date is 02/14/2017

We have successfully created funnelcakes before bundled with an add-on in https://bugzilla.mozilla.org/show_bug.cgi?id=1195843, https://bugzilla.mozilla.org/show_bug.cgi?id=1272776 and https://bugzilla.mozilla.org/show_bug.cgi?id=1314629.

There will be two versions of the add-on this time (one in EN and one in DE, so these will be 


EN:
funnelcake 101 and 102 (https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0)

DE: 
funnelcake 103 and 104


Variations: 

Funnelcake 101: EN control -- totally vanilla Firefox 51 with just the configs to denote the funnelcake

Funnelcake 102: EN onboarding add-on + Firefox 51 with auto-import code that was created in bugs 1271775 1271799.


Funnelcake 103: DE control -- totally vanilla Firefox 51 with just the configs to denote the funnelcake

Funnelcake 104: DE onboarding add-on + Firefox 51 with auto-import code that was created in bugs 1271775 1271799.

Comment 1

[Ben Niolet]

Hi folks. Can you view this request above? The same as the last few onboarding funnelcakes.

Note that we're still trying to dial in our launch date. It won't be before Feb. 14, but may slip another week.

Comment 2

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

(In reply to Ben Niolet from comment #1)
> Hi folks. Can you view this request above? The same as the last few
> onboarding funnelcakes.
> 
> Note that we're still trying to dial in our launch date. It won't be before
> Feb. 14, but may slip another week.

Hi Ben, thanks for the heads-up. I am a bit concerned about the start date of this FC experiment/build. 
* Fx51 goes live Jan 24th
* FC 101-104 (based on fx51) gets pushed Feb 14/21st
* Fx 52 goes live March 7th.

Are you ok with your FC experiment running only for 2-3 weeks? Will you be able to meet all the goals behind the FC experiment? I am assuming once 52 goes live and updates are enabled at 100% all 51 builds including FC builds are auto-updated to Fx52. Right?

Comment 3

[Chris More [:cmore]]

(In reply to Ritu Kothari (:ritu) from comment #2)
> (In reply to Ben Niolet from comment #1)
> > Hi folks. Can you view this request above? The same as the last few
> > onboarding funnelcakes.
> > 
> > Note that we're still trying to dial in our launch date. It won't be before
> > Feb. 14, but may slip another week.
> 
> Hi Ben, thanks for the heads-up. I am a bit concerned about the start date
> of this FC experiment/build. 
> * Fx51 goes live Jan 24th
> * FC 101-104 (based on fx51) gets pushed Feb 14/21st
> * Fx 52 goes live March 7th.
> 
> Are you ok with your FC experiment running only for 2-3 weeks? Will you be
> able to meet all the goals behind the FC experiment? I am assuming once 52
> goes live and updates are enabled at 100% all 51 builds including FC builds
> are auto-updated to Fx52. Right?

Hi Ritu.

The plan is to keep v1.6 in-market until one of two conditions is true:

1) When the Firefox product team has something that beats it on a key product metric

2) If data comes back that shows that it doesn't improve retention (we need 3-6 weeks of data after launch).

For the #1 item above, it could be into Q2 until there is something from the product team that could beat it, so that means the v1.6 will be crossing over the 52.0 and 53.0 window and possibly 54.0. So, that means that the v1.6 funnelcake will need to be re-spin these funnelcakes once 52.0 comes and the bouncer entries updated. These are unlike other funnelcakes that are normally just 2 weeks long and thus these funnelcakes will stay live until one of the two conditions above are true.

Comment 4

[Liz Henry (:lizzard) (relman/hg->git project)]

As long as this has QE lined up and a test plan and QE signs off, I'm OK with this.

Comment 5

[Chris More [:cmore]]

Updating launch of v1.6 to 2/21/2017

Comment 6

[Chris More [:cmore]]

Ritu/Nthomas: Just want to confirm that we're all on the same page that v1.6 will stay in-market until we have data back on the conclusion. This is different than the past where we run them for only 2 weeks. This means the funnelcake will be running while Firefox 52 comes out and maybe even Firefox 53. So, that means these 3 funnelcakes will need to be re-spun (with the XPI) with Firefox 52 after March 7th and the bouncer product entry will need to be updated when Firefox 52 is released.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

I will have to look into how much we can automate funnelcake builds as part of the release pipeline, to avoid donkey work and serving old builds as we create new release. We can likely create the builds fairly easily, not sure about the bouncer update part yet. Could you please confirm what bouncer products you're expecting as firefox version changes. Any ETA on first request for builds ?

Regarding the funnelcake id numbers, we could easily support using 101 as control for both en-US and de, and 102 for the test. It would actually be simpler for us to write the configuration, if there are no other considerations on the distribution and analysis side.

Comment 8

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Thanks Chris! I understand. FYI Julien (52 owner) and Liz (53 owner).

Comment 9

[Julien Cristau [:jcristau]]

Thanks for the heads-up.

Comment 10

[Schalk Neethling [:espressive]]

Also, please note that the funnelcake needs to be built with the following flag set to false ~ browser.newtab.preload

Comment 11

[Chris More [:cmore]]

(In reply to Schalk Neethling [:espressive] from comment #10)
> Also, please note that the funnelcake needs to be built with the following
> flag set to false ~ browser.newtab.preload

Copy that. 

nthomas: FYI at comment 10 for funnelcake 102 (en-us variation) and funnelcake 104 (de-DE variations) to make sure we have that flag set to false.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

cmore: OK. Did you see comment #7 ?

Comment 13

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #7)
> I will have to look into how much we can automate funnelcake builds as part
> of the release pipeline, to avoid donkey work and serving old builds as we
> create new release. We can likely create the builds fairly easily, not sure
> about the bouncer update part yet. Could you please confirm what bouncer
> products you're expecting as firefox version changes. Any ETA on first
> request for builds ?

Should be:

https://download.mozilla.org/?product=firefox-stub-f101&os=win&lang=en-US 
https://download.mozilla.org/?product=firefox-stub-f102&os=win&lang=en-US 
https://download.mozilla.org/?product=firefox-stub-f103&os=win&lang=de
https://download.mozilla.org/?product=firefox-stub-f104&os=win&lang=de

Given that the Firefox version is not in the bouncer product name, it just means the bounce redirect to the correct version of Firefox should be updated once a new version of Firefox is released. This should keep the URLs valid over longer periods of time and not create any 404s.

> 
> Regarding the funnelcake id numbers, we could easily support using 101 as
> control for both en-US and de, and 102 for the test. It would actually be
> simpler for us to write the configuration, if there are no other
> considerations on the distribution and analysis side.

Yes, possible for sure given also how our websites work. It would just complicate the reporting on the other side. I'd rather just keep the two controls separate from each other, so we don't run into any unforeseen issues.

Comment 14

[Schalk Neethling [:espressive]]

(In reply to Chris More [:cmore] from comment #13)
> (In reply to Nick Thomas [:nthomas] from comment #7)
> > I will have to look into how much we can automate funnelcake builds as part
> > of the release pipeline, to avoid donkey work and serving old builds as we
> > create new release. We can likely create the builds fairly easily, not sure
> > about the bouncer update part yet. Could you please confirm what bouncer
> > products you're expecting as firefox version changes. Any ETA on first
> > request for builds ?
> 
> Should be:
> 
> https://download.mozilla.org/?product=firefox-stub-f101&os=win&lang=en-US 
> https://download.mozilla.org/?product=firefox-stub-f102&os=win&lang=en-US 
> https://download.mozilla.org/?product=firefox-stub-f103&os=win&lang=de
> https://download.mozilla.org/?product=firefox-stub-f104&os=win&lang=de
> 
> Given that the Firefox version is not in the bouncer product name, it just
> means the bounce redirect to the correct version of Firefox should be
> updated once a new version of Firefox is released. This should keep the URLs
> valid over longer periods of time and not create any 404s.
> 
> > 
> > Regarding the funnelcake id numbers, we could easily support using 101 as
> > control for both en-US and de, and 102 for the test. It would actually be
> > simpler for us to write the configuration, if there are no other
> > considerations on the distribution and analysis side.
> 
> Yes, possible for sure given also how our websites work. It would just
> complicate the reporting on the other side. I'd rather just keep the two
> controls separate from each other, so we don't run into any unforeseen
> issues.

So, does this mean we will only use 101 as the distribution.id and not have to distinguish between German and US/Canada?

Comment 15

[Chris More [:cmore]]

Attachment: firefox_all_aboard_16-1.6.1-fx.xpi (initial self-signed version)

nthomas: please use this XPI for funnelcake 102 (en-US) and funnelcake 104 (de) onboarding builds.

Also, please note comment 10 for the specific config needed for both of those funnelcakes. Thanks!

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Initial configuration landed at https://github.com/mozilla-partners/funnelcake/commit/5e646e2c90e927fa81b7e1288eded206f0d0ff83. I've left out 'browser.migrate.automigrate.ui.enabled=false' in 102 & 104 - we had it in funnelcake94 (bug 1314629), but not in funnelcake99 or 100 (bug 1326252). Let me know if I did that three-way merge incorrectly and it should be re-added.

funnelcake101:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake101/v1/win32/en-US/Firefox%20Setup%2051.0.1.exe
SHA1: 6779e400682c1c480a57d67df722e0065deec29c

funnelcake102:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake102/v1/win32/en-US/Firefox%20Setup%2051.0.1.exe
SHA1: bb00b03f034ed1371e000fd35f8dd0f4e963443b

funnelcake103:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake103/v1/win32/de/Firefox%20Setup%2051.0.1.exe
SHA1: 48596b9f46d315913da576fd7c0f9fcb30a692ea

funnelcake104:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake104/v1/win32/de/Firefox%20Setup%2051.0.1.exe
SHA1: fd44ba79131285be927ed34945f34d8d74df1587

Status - builds ready for checking. They're signed but I'm expecting a respin for the addon. Bouncer set up is a TODO.

Comment 17

[Schalk Neethling [:espressive]]

:cmore - Please have a look at my question in https://bugzilla.mozilla.org/show_bug.cgi?id=1332726#c14 - Thanks

Comment 18

[Chris More [:cmore]]

(In reply to Schalk Neethling [:espressive] from comment #14)
> (In reply to Chris More [:cmore] from comment #13)
> > (In reply to Nick Thomas [:nthomas] from comment #7)
> > > I will have to look into how much we can automate funnelcake builds as part
> > > of the release pipeline, to avoid donkey work and serving old builds as we
> > > create new release. We can likely create the builds fairly easily, not sure
> > > about the bouncer update part yet. Could you please confirm what bouncer
> > > products you're expecting as firefox version changes. Any ETA on first
> > > request for builds ?
> > 
> > Should be:
> > 
> > https://download.mozilla.org/?product=firefox-stub-f101&os=win&lang=en-US 
> > https://download.mozilla.org/?product=firefox-stub-f102&os=win&lang=en-US 
> > https://download.mozilla.org/?product=firefox-stub-f103&os=win&lang=de
> > https://download.mozilla.org/?product=firefox-stub-f104&os=win&lang=de
> > 
> > Given that the Firefox version is not in the bouncer product name, it just
> > means the bounce redirect to the correct version of Firefox should be
> > updated once a new version of Firefox is released. This should keep the URLs
> > valid over longer periods of time and not create any 404s.
> > 
> > > 
> > > Regarding the funnelcake id numbers, we could easily support using 101 as
> > > control for both en-US and de, and 102 for the test. It would actually be
> > > simpler for us to write the configuration, if there are no other
> > > considerations on the distribution and analysis side.
> > 
> > Yes, possible for sure given also how our websites work. It would just
> > complicate the reporting on the other side. I'd rather just keep the two
> > controls separate from each other, so we don't run into any unforeseen
> > issues.
> 
> So, does this mean we will only use 101 as the distribution.id and not have
> to distinguish between German and US/Canada?

101 = English, 104 = German. Nick is setting the distribution.id unique for each of the cohort set: 101/102 +en, 103/104=de. Both will have the same xpi because it appears the 1.6 branch has both strings in one add+on. Is code specific distribution.id hardcoded into the xpi or does it just append?

Comment 19

[Chris More [:cmore]]

Nthomas: yes. There will be multiple re spins and one at a minimum when the final xpi is amo approved.

Comment 20

[Schalk Neethling [:espressive]]

(In reply to Chris More [:cmore] from comment #18)
> (In reply to Schalk Neethling [:espressive] from comment #14)
> > (In reply to Chris More [:cmore] from comment #13)
> > > (In reply to Nick Thomas [:nthomas] from comment #7)
> > > > I will have to look into how much we can automate funnelcake builds as part
> > > > of the release pipeline, to avoid donkey work and serving old builds as we
> > > > create new release. We can likely create the builds fairly easily, not sure
> > > > about the bouncer update part yet. Could you please confirm what bouncer
> > > > products you're expecting as firefox version changes. Any ETA on first
> > > > request for builds ?
> > > 
> > > Should be:
> > > 
> > > https://download.mozilla.org/?product=firefox-stub-f101&os=win&lang=en-US 
> > > https://download.mozilla.org/?product=firefox-stub-f102&os=win&lang=en-US 
> > > https://download.mozilla.org/?product=firefox-stub-f103&os=win&lang=de
> > > https://download.mozilla.org/?product=firefox-stub-f104&os=win&lang=de
> > > 
> > > Given that the Firefox version is not in the bouncer product name, it just
> > > means the bounce redirect to the correct version of Firefox should be
> > > updated once a new version of Firefox is released. This should keep the URLs
> > > valid over longer periods of time and not create any 404s.
> > > 
> > > > 
> > > > Regarding the funnelcake id numbers, we could easily support using 101 as
> > > > control for both en-US and de, and 102 for the test. It would actually be
> > > > simpler for us to write the configuration, if there are no other
> > > > considerations on the distribution and analysis side.
> > > 
> > > Yes, possible for sure given also how our websites work. It would just
> > > complicate the reporting on the other side. I'd rather just keep the two
> > > controls separate from each other, so we don't run into any unforeseen
> > > issues.
> > 
> > So, does this mean we will only use 101 as the distribution.id and not have
> > to distinguish between German and US/Canada?
> 
> 101 = English, 104 = German. Nick is setting the distribution.id unique for
> each of the cohort set: 101/102 +en, 103/104=de. Both will have the same xpi
> because it appears the 1.6 branch has both strings in one add+on. Is code
> specific distribution.id hardcoded into the xpi or does it just append?

I am looking at that now. Now that I am sure that there will be two, I need to see whether I have access to the locale from within the add-on code, else I can use the lang files as a hacky workaround.

Comment 21

[Chris More [:cmore]]

nthomas: we are making a modification for funnelcake 102 and 104 (en-us and de variations with XPI), we are going to disable the auto-migration given bug 1249008.

We will have a new XPI soon that will disable the about:newtab modification, but we will need you to adjust the config for 102 and 104 to *disable* or not enable auto migration for those builds. 101 through 104 will get the manual migration window.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

I've started a pull request with browser.migrate.automigrate.enabled=true removed for 102 and 104 (see https://github.com/mozilla-partners/funnelcake/pull/50). Please advise if you still need browser.newtab.preload=false if you're no longer modifying about:newtab. 

Waiting on the new XPI before generating builds.

Comment 23

[Schalk Neethling [:espressive]]

(In reply to Nick Thomas [:nthomas] from comment #22)
> I've started a pull request with browser.migrate.automigrate.enabled=true
> removed for 102 and 104 (see
> https://github.com/mozilla-partners/funnelcake/pull/50). Please advise if
> you still need browser.newtab.preload=false if you're no longer modifying
> about:newtab. 

Nope, we will no longer need it.

> 
> Waiting on the new XPI before generating builds.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Refreshed PR 50 to drop browser.newtab.preload.

Comment 25

[Chris More [:cmore]]

Attachment: firefox_all_aboard_16-1.6.3-fx.xpi (latest self-signed version)

:nthomas: here's the latest XPI to be added to funnelcake 102 and 104. Please make sure the auto migration flags are flipped off and don't need to flip the preload to false either.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Configuration updated at https://github.com/mozilla-partners/funnelcake/commit/284b30702967d30f0ce375013a27748355651ef5 with all requests. v2 builds generated...

funnelcake101:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake101/v2/win32/en-US/Firefox%20Setup%2051.0.1.exe
SHA1: 25d25b94ff3f650abebfa9ea798fac37deb06ba4

funnelcake102:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake102/v2/win32/en-US/Firefox%20Setup%2051.0.1.exe
SHA1: 51d769a28c85da6a32bf5c8fc3019a33909a4bc8

funnelcake103:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake103/v2/win32/de/Firefox%20Setup%2051.0.1.exe
SHA1: b719f09ea9737be249e919a8df1206ab692b087a

funnelcake104:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake104/v2/win32/de/Firefox%20Setup%2051.0.1.exe
SHA1: b02fd1e9e4c666ba0dffbdab1c0d6fcc189a0f98

Status - builds ready for checking. They're signed but I'm expecting a respin for the addon. Bouncer set up is a TODO.

Comment 27

[Jenny Douglas [:JennyD]]

Hi Grover, we will ready for QA to review these funnelcakes once they are created. Please let me know if you need any other info.

Comment 28

[u554753]

Hi Jenny, did a quick lookover on this bug. Are there any other differences between the four builds other than what is noted on Comment 0?

Comment 29

[u554753]

Also, what kind of QA are you looking for in this regard? Do you want a full end-to-end, or just to verify that the different builds automigrate/manual migrate? 

Also passing this onto fellow QA colleague Justin Williams.

Comment 30

[Jenny Douglas [:JennyD]]

Hi Grover, Hi Grover, There are no other differences with the four builds other than what Ben noted in the first comment. 
We are looking to make sure that is works with firefox, so I don't think we need real end-to-end. 
Hope this helps.
Thanks,
Jenny

Comment 31

[Jenny Douglas [:JennyD]]

Hi Grover,
Wanted to check in with you on the QA testing to make sure 1.6 works with firefox. We have completed our internal testing. 
We are ready to release once you sign off.

Thanks,
Jenny

Comment 32

[Justin [:JW_SoftvisionQA]]

I can verify 1.6 works with firefox as intended.

Comment 33

[Chris More [:cmore]]

(In reply to Justin [:JW_SoftvisionQA] from comment #32)
> I can verify 1.6 works with firefox as intended.

Great!

Thanks

Comment 34

[Chris More [:cmore]]

:espressive: Can you verify that the 1.6 repo has all of the latest changes in it and nothing has come up in any testing? If good, please increment the counter to .4 (had to update it to .3 last week on the repo) and please merge. 

next: uploading the XPI to AMO for approval.

Thanks!

Comment 35

[Schalk Neethling [:espressive]]

:cmore - Done

Comment 36

[Chris More [:cmore]]

v1.6 has been uploaded to AMO for review:

https://addons.mozilla.org/en-US/developers/addon/firefox-all-aboard-1-6/edit

After it has been reviewed, I will download the signed XPI and :nthomas can use that to make the final funnelcake.

Comment 37

[Chris More [:cmore]]

Attachment: firefox_all_aboard_16-1.6.4-fx.xpi (final v1.6 XPI that is AMO signed)

We're ready! AMO approved and signed XPI.

:nthomas: please use this XPI in both the final funnelcake 102 and 104.

1) Use this latest XPI
2) Set up production bouncer links to all 4 funnelcakes 

Then I will:

1) test full funnel
2) Test each links work
3) Create optimizely test for redirects
4) Get green light and launch

Thanks!

Comment 38

[Nick Thomas [:nthomas] (UTC+12)]

Funnelcake configuration updated in https://github.com/mozilla-partners/funnelcake/commit/6875bf950e6aebf5472686b2a03ca477670f311e. v3 builds generated...

funnelcake101:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake101/v3/win32/en-US/Firefox%20Setup%2051.0.1.exe
SHA256: 8eff0a4e429e5394e5c5347b6e0e3bb67dd80b5e1f5029ad3d9d4168dddd704c

funnelcake102:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake102/v3/win32/en-US/Firefox%20Setup%2051.0.1.exe
SHA256: 6f8c5164e6696d46ca2e0012730cb9b220dfd2467a345f3a5b50887aeb7ac41e

funnelcake103:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake103/v3/win32/de/Firefox%20Setup%2051.0.1.exe
SHA256: d8aeb9571a56108602e9bdf1c55c1e4e46d6fb1d4a5b734f3eaa18994fbbe61e

funnelcake104:
https://archive.mozilla.org/pub/firefox/candidates/51.0.1-candidates/build3/partner-repacks/funnelcake104/v3/win32/de/Firefox%20Setup%2051.0.1.exe
SHA256: 1d6cdb357bffb6304ca10e89f7202e2d70988308e8356a0c575c36413b85d14d

Bouncer set up pointing into the candidates builds (to enable end-to-end testing):

https://download.mozilla.org/?product=firefox-stub-f101&os=win&lang=en-US
https://download.mozilla.org/?product=firefox-stub-f102&os=win&lang=en-US
https://download.mozilla.org/?product=firefox-stub-f103&os=win&lang=de
https://download.mozilla.org/?product=firefox-stub-f104&os=win&lang=de

Status: ready for testing. Once we're happy I'll move the installers into firefox/releases/ and change the bouncer locations to point there.

Comment 39

[Chris More [:cmore]]

Thanks, nthomas!

Jenny:

We need the team to do a final test to make sure these links download the correct installer and that onboarding is triggers:

https://download.mozilla.org/?product=firefox-stub-f101&os=win&lang=en-US (control)
https://download.mozilla.org/?product=firefox-stub-f102&os=win&lang=en-US (en onboarding)
https://download.mozilla.org/?product=firefox-stub-f103&os=win&lang=de (control)
https://download.mozilla.org/?product=firefox-stub-f104&os=win&lang=de (de onboarding)

Comment 40

[Schalk Neethling [:espressive]]

Works as expected for the most part. One thing to note: With the en-US version I get the import wizard but, with the DE version I do not. Perhaps this is intentional, but wanted to call it out in case it is a configuration issue with the DE builds. Other than that, all looks good.

Comment 41

[Ben Niolet]

Attachment: German import wizard

Schalk,

Can you double check? When I tested, the import wizard fired for both DE and EN builds.

Comment 42

[Chris More [:cmore]]

(In reply to Ben Niolet from comment #41)
> Created attachment 8842857 [details]
> German import wizard
> 
> Schalk,
> 
> Can you double check? When I tested, the import wizard fired for both DE and
> EN builds.

Also, Ben/Schalk: can you confirm that there is no about:newtab modification after you confirm the migration wizard opens for all builds?

Comment 43

[Schalk Neethling [:espressive]]

(In reply to Ben Niolet from comment #41)
> Created attachment 8842857 [details]
> German import wizard
> 
> Schalk,
> 
> Can you double check? When I tested, the import wizard fired for both DE and
> EN builds.

Will do, probably because I did not kill the profile entirely in the file system.

Comment 44

[Schalk Neethling [:espressive]]

(In reply to Chris More [:cmore] from comment #42)
> (In reply to Ben Niolet from comment #41)
> > Created attachment 8842857 [details]
> > German import wizard
> > 
> > Schalk,
> > 
> > Can you double check? When I tested, the import wizard fired for both DE and
> > EN builds.
> 
> Also, Ben/Schalk: can you confirm that there is no about:newtab modification
> after you confirm the migration wizard opens for all builds?

Will double check

Comment 45

[Schalk Neethling [:espressive]]

This is where I am at the moment with testing:

https://download.mozilla.org/?product=firefox-stub-f104&os=win&lang=de

Import Wizard: Yes
Onboarding triggered: Yes
No newtab modifications: Yes

https://download.mozilla.org/?product=firefox-stub-f102&os=win&lang=en-US

Import Wizard: Yes
Onboarding triggered: Yes
No newtab modifications: Yes

Comment 46

[Chris More [:cmore]]

(In reply to Schalk Neethling [:espressive] from comment #45)
> This is where I am at the moment with testing:
> 
> https://download.mozilla.org/?product=firefox-stub-f104&os=win&lang=de
> 
> Import Wizard: Yes
> Onboarding triggered: Yes
> No newtab modifications: Yes
> 
> https://download.mozilla.org/?product=firefox-stub-f102&os=win&lang=en-US
> 
> Import Wizard: Yes
> Onboarding triggered: Yes
> No newtab modifications: Yes

Great! Looks like we are all good.

:nthomas: we are good to go. Anything else you need to do on your side?

Comment 47

[Schalk Neethling [:espressive]]

https://download.mozilla.org/?product=firefox-stub-f103&os=win&lang=de

Import Wizard: Yes
Onboarding triggered: No
Addon present: No
No newtab modifications: N/A [Yes]

https://download.mozilla.org/?product=firefox-stub-f101&os=win&lang=en-US

Import Wizard: Yes
Onboarding triggered: No
Addon present: No
No newtab modifications: N/A [Yes]

Comment 48

[Chris More [:cmore]]

:nthomas: Given that we want to launch 1.6 today and that 1.6 will stay in market much longer than previous onboarding experiments, it will need to be re-spun as Fx52 next week after 52.0 is out the door.

Is your plan to launch today with Fx51 and then after the 52 release, you will re-sin these funnelcakes using Fx52 and update the bouncer redirects to point to the correct Fx52 exes?

Comment 49

[Chris More [:cmore]]

Update: Started 1.6 experiment on 3/2/2017 at 11:00am PST at 20% volume. Will go up to 100% volume this Saturday night.

Comment 50

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Chris More [:cmore] from comment #46)
> :nthomas: we are good to go. Anything else you need to do on your side?

I moved the files to a durable location
 firefox-stub-f101, /firefox/releases/51.0.1-funnelcake101/win32/en-US/Firefox Setup 51.0.1.exe
 firefox-stub-f102, /firefox/releases/51.0.1-funnelcake102/win32/en-US/Firefox Setup 51.0.1.exe 
 firefox-stub-f103, /firefox/releases/51.0.1-funnelcake103/win32/de/Firefox Setup 51.0.1.exe
 firefox-stub-f104, /firefox/releases/51.0.1-funnelcake104/win32/de/Firefox Setup 51.0.1.exe
and updated bouncer, then verified the links work and match the hashes in comment #38. Co-ordinated with cmore on IRC to pause the experiment while we swapped over.


(In reply to Chris More [:cmore] from comment #48)
> Is your plan to launch today with Fx51 and then after the 52 release, you
> will re-sin these funnelcakes using Fx52 and update the bouncer redirects to
> point to the correct Fx52 exes?

I'll make sure we have builds generated today, that bits are published next week, and that the release engineer on the hook for 52.0 knows how to update bouncer at release time.

Comment 51

[Nick Thomas [:nthomas] (UTC+12)]

New v3 builds generated using Firefox 52.0 build2, which ships Tuesday:

funnelcake101:
https://archive.mozilla.org/pub/firefox/candidates/52.0-candidates/build2/partner-repacks/funnelcake101/v3/win32/en-US/Firefox%20Setup%2052.0.exe
SHA256: 4310aa58466bad0b84941a5293ce2c448b8231195d88c8e3c098008c59f00dbb

funnelcake102:
https://archive.mozilla.org/pub/firefox/candidates/52.0-candidates/build2/partner-repacks/funnelcake102/v3/win32/en-US/Firefox%20Setup%2052.0.exe
SHA256: da51b3e15d28dcc0d4e63c8f8db7e327e68a3c52e1fce42604ee9a374f204146

funnelcake103:
https://archive.mozilla.org/pub/firefox/candidates/52.0-candidates/build2/partner-repacks/funnelcake103/v3/win32/de/Firefox%20Setup%2052.0.exe
SHA256: 1d447aad44eb851650f0c00c0d85f33cce91dfb8d4b1f6303b4dc9c65f07ebf5

funnelcake104:
https://archive.mozilla.org/pub/firefox/candidates/52.0-candidates/build2/partner-repacks/funnelcake104/v3/win32/de/Firefox%20Setup%2052.0.exe
SHA256: b16a03466f98eb4e62a266d7a0c32aead2493dac480a5a303af6dce2b540d198

cmore, what sort of QA protocol do you want to follow as new Firefox versions come along ?

Comment 52

[Nick Thomas [:nthomas] (UTC+12)]

Filed bug 1345000 to investigate automating these builds.

Comment 53

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Doc on funnelcake generation, push, bouncer update

I did a quick spot-check of the builds in comment #51 - the firstrun pages were all fine, addon was present in 102 & 104 and modified the first run page, signing was OK. Tempting to push on that basis but don't need to until early PST tomorrow, which gives cmore a chance to chime in.

This attachment describes the process (for the benefit of jlorenzo, jlund, and other releng). Steps 2 and 3 to do for 52.0.

Comment 54

[Chris More [:cmore]]

All good! Just ping me when the new versions are live and I will double check Google Analytics that I can see the full funnel.

Comment 55

[Jordan Lund (:jlund)]

ftr - the new funnelcakes for 101-104 are in the releases dir:

http://archive.mozilla.org/pub/firefox/releases/51.0.1-funnelcake101/win32/en-US/Firefox%20Setup%2051.0.1.exe
http://archive.mozilla.org/pub/firefox/releases/51.0.1-funnelcake102/win32/en-US/Firefox%20Setup%2051.0.1.exe
http://archive.mozilla.org/pub/firefox/releases/51.0.1-funnelcake103/win32/de/Firefox%20Setup%2051.0.1.exe
http://archive.mozilla.org/pub/firefox/releases/51.0.1-funnelcake104/win32/de/Firefox%20Setup%2051.0.1.exe

just need to update bouncer

Comment 56

[Jordan Lund (:jlund)]

derp, those should have been 52.0 locations url links. paste fail.

just waiting for them to show up on archive.m.o then I'll post them

Comment 57

[Jordan Lund (:jlund)]

jlund> Jordan Lund cmore: bouncer is updated should be live. sentry probably still has to run
14:24:03 so another couple min and we should be confident
14:24:09 
<cmore> 10-4
14:24:43 
<•nthomas> looks like the redirects have changed, checking hashes now
14:25:17 
14:26:46 
<cmore> nthomas jlund: restarting experiment and will monitor real-time data
14:27:09 
<•nthomas> hashes are OK
14:28:09 
<cmore> nthomas jlund: funnelcake downloads are coming through. Waiting for first runs to start showing up

we are live and new hashes match comment 51. cmore is sanity checking results

Comment 58

[Chris More [:cmore]]

All is good with 52.0! Thanks releng.

Comment 59

[Chris More [:cmore]]

One quick update. There was a redirect issue on mozilla.org for the de funnelcake and that was resolved today, which means the DE build is going out the door now. Prior to this change, the non-funnelcake DE build was being distributed. I will update the funnelcake master tracker.

Comment 60

[Chris More [:cmore]]

Update: We are going to pause the funnelcakes on March 26th for two reasons:

1) Need to run a heartbeat survey that requires the funnelcakes to be stopped.
2) To wait for retention data to come in by April 12th.

On April 12th, we should have both retention, heartbeat and other metrics on 1.6 vs the control.

On April 12th or soon after we can decide to re-enable 1.6 funnelcakes, not-restart, or made a change to 1.6

I've updated Optimizely to auto-shutoff on March 26th at 8am.

Comment 61

[Nick Thomas [:nthomas] (UTC+12)]

Could we please suspend this ASAP while we ship 52.0.1. Same for bug 1345993 if it's live.

Comment 62

[Chris More [:cmore]]

Paused. Please ping me on IRC and let me know when I an re-enable.

Comment 63

[Nick Thomas [:nthomas] (UTC+12)]

Regenerated for 52.0.1, builds pushed and bouncer updated. Co-ordinated with cmore on IRC to re-enable.

Comment 64

[Chris More [:cmore]]

Funnelcakes re-started on 3:11pm pst.

Comment 65

[Nick Thomas [:nthomas] (UTC+12)]

Will we retire this in favor of bug 1348127, or do we need the data on April 12th first ?

Comment 66

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #65)
> Will we retire this in favor of bug 1348127, or do we need the data on April
> 12th first ?

Not really. We need the data starting next week to decide if 1.6 will be turned on again. I wouldn't do any extra work for 1.6 until we have the retention data back. Stay tuned!

Comment 67

[Nick Thomas [:nthomas] (UTC+12)]

Any news here ? I'm retiring the configs but easy to reverse if still needed.

Comment 68

[Nick Thomas [:nthomas] (UTC+12)]

Please reopen if we want to restart this.