Closed Bug 1314629 Opened 8 years ago Closed 7 years ago

Funnelcakes 93/94 50.0.1 for Onboarding Add-on Experiment (11/23/2016 launch)

Categories

(Release Engineering :: Release Requests, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bniolet, Assigned: nthomas)

References

Details

Attachments

(1 file, 5 obsolete files)

We are building a new version of the onboarding add-on experiment. (https://github.com/mozilla/all-aboard)

The experiment will launch with a version of Firefox bundled with the Version 1.5 Add-on XPI. Our target launch date is 11/16/2016

We have successfully created funnelcakes before bundled with an add-on in https://bugzilla.mozilla.org/show_bug.cgi?id=1195843 and https://bugzilla.mozilla.org/show_bug.cgi?id=1272776.

These will be funnelcake 93 and 94 (https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0)


Variations: 

Funnelcake 93: control -- totally vanilla Firefox 50.0.1 with just the configs to denote the funnelcake

Funnelcake 94: onboarding add-on + Firefox 50.0.1 with auto-import code that was created in bugs 1271775 1271799.
Thank, Ben! 

:nthomas: can you view this request above and let me know if you have any issues with it? The same as the last few onboarding funnelcakes.
Flags: needinfo?(nthomas)
:ritu :lizzard ^^ check out the above request for the next iteration and let me know if you have feedback.
Flags: needinfo?(rkothari)
Flags: needinfo?(lhenry)
Hi Chris, a few things:
1. When will QA test this?
2. The start date of 11/16 is a day after 50 go-live (11/15). In the past, we've generally done these funnelcake experiments at least a week after the mainline release is live. Isn't that right? This delay gives us enough time to determine if there are any severe crashes/hangs/regressions plague 50. 
3. When will this experiment end?
4. What %age of release users is this experiment going to be rolled out to? Thanks!
Flags: needinfo?(rkothari)
Based on the spreadsheet, this is en-US for win32 only. As an extension of previous onboarding funnelcakes this should be no problem, as all the automigration code is already present in 49. I'll reiterate Ritu's comment about shipping only one day after 50.0, so the 50.0.1 expectation isn't right.

We already have RC builds for 50.0, so builds can be generated as soon as you have an XPI worth testing.
Flags: needinfo?(nthomas)
See Also: → 1312666
QA Update: Chris SV-Vegas office can assist if testing will be needed.
Flags: needinfo?(lhenry)
Flags: needinfo?(chrismore.bugzilla)
(In reply to Ritu Kothari (:ritu) from comment #3)
> Hi Chris, a few things:
> 1. When will QA test this?

We should be ready for QA next week. We are wrapping up our team's QA now. Most of the experience is similar to the previous v1.2 experiment and we have a new design and one less step. QA test plan is in the works.

> 2. The start date of 11/16 is a day after 50 go-live (11/15). In the past,
> we've generally done these funnelcake experiments at least a week after the
> mainline release is live. Isn't that right? This delay gives us enough time
> to determine if there are any severe crashes/hangs/regressions plague 50. 

The 16th was the original date before Fx 50 was pushed a week out, so we should also push out too. Nov 22nd?

> 3. When will this experiment end?

The experiment needs to run a full two weeks from Sunday through Saturday. So, if it starts on the 22nd (partial week), the clock starts on the full two weeks on the 27th and runs through Dec 10th. 

> 4. What %age of release users is this experiment going to be rolled out to?

About 80% of English+United States+Win32 visitors downloading from www.mozilla.org during the two weeks. That will get the cohorts big enough to get statistical significance. 

> Thanks!
Flags: needinfo?(chrismore.bugzilla)
Attached file all-aboard-v1-5-130.xpi (obsolete) —
Here is the latest version of the add-on xpi for the 1.5 onboarding funnelcake. 

We are ready for a build for our team to start internal QA.
Flags: needinfo?(coop)
(In reply to Ben Niolet from comment #7)
> We are ready for a build for our team to start internal QA.

Redirecting to Nick.
Flags: needinfo?(coop) → needinfo?(nthomas)
I'll generate some builds.
Assignee: nobody → nthomas
Flags: needinfo?(nthomas)
Attached file New (signed) xpi for add-on v. 1.5 (obsolete) —
Here is an updated xpi (this one is signed). Can you please build a new version of 94 using this? 

Also we wanted the auto-import to be included with this. (I believe that's a pref?)
Attachment #8808276 - Attachment is obsolete: true
Flags: needinfo?(nthomas)
Config updated at https://github.com/mozilla-partners/funnelcake/commit/f357b2f6b631d0d250638a117dfa515341d76472. FYI, the XPI does not include a config.json to adjust the event timing. 

Automigrate was already set in v1, and works for me with the build below. Did you rename/remove %APPDATA%\Mozilla\Firefox to simulate a new profile ?

Updated 94:
https://archive.mozilla.org/pub/firefox/candidates/50.0-candidates/build2/partner-repacks/funnelcake94-testing/v2/win32/en-US/Firefox%20Setup%2050.0.exe (SHA1: 44e22791e216bf683d8557170564adb07fb33316)
Flags: needinfo?(nthomas)
Thanks, nthomas. Not sure why auto-import was failing. Working properly now. We're testing away.
Summary: Funnelcakes 93/94 50.0.1 for Onboarding Add-on Experiment (11/16/2016 launch) → Funnelcakes 93/94 50.0.1 for Onboarding Add-on Experiment (11/23/2016 launch)
Blocks: 1317751
Summary: Funnelcakes 93/94 50.0.1 for Onboarding Add-on Experiment (11/23/2016 launch) → Funnelcakes 93/94 50.0.1 for Onboarding Add-on Experiment (11/22/2016 launch)
nthomas: we should have the final XPI tomorrow to build the final funnelcake. Will it be 50.0 or 50.0.1?

Michelle: Once nthomas the latest funnelcake, you should be able to test the final build. Most of the mechanics of the add-on is the same with v1.5. Just some new visuals, new copy in the side bar and one less sidebar. The rest is basically the same.

Scheduled release: 11/22/2016
Flags: needinfo?(nthomas)
Flags: needinfo?(mfunches)
50.0 has been out for a few hours, and we release the updates slowly at first, so it's too soon to tell if we'll be on 50.0 or 50.0.1 by 22 November. Ritu is on point for Release Management if you need someone to ask.

I'll generate with the final XPI when it's ready, and regenerate if we do need 50.0.1. Holding off on bouncer etc until we know for sure.
Flags: needinfo?(nthomas)
Thanks for the update Chris. I will start prep for test.
Flags: needinfo?(mfunches)
I will reiterate that Funnelcake experiments should be pushed out ~a week after the mainline release has been live on release channel. We are investigating a few issues at the moment on 50 release build. I will have a more firm handle on dot release drivers by Friday (at the earliest). We should plan to build and push the week of 11/21 assuming there are no dot release drivers.
(In reply to Ritu Kothari (:ritu) from comment #17)
> I will reiterate that Funnelcake experiments should be pushed out ~a week
> after the mainline release has been live on release channel. We are
> investigating a few issues at the moment on 50 release build. I will have a
> more firm handle on dot release drivers by Friday (at the earliest). We
> should plan to build and push the week of 11/21 assuming there are no dot
> release drivers.

Yes, I agree. That's why this experiment was moved to the week of 11/21 because the mainline release of Firefox was pushed back a week too. Launching on the 16th was when Firefox was supposed to be release last week. Even though the plan is to launch next week, retention won't start to be measured until 11/27 because we need a full Sunday through Saturday per week for proper analysis.
Attached file all_aboard-1.5.1-fx.xpi (obsolete) —
Here is the final xpi. Can you spin up a funnelcake with it, please?
Attachment #8808781 - Attachment is obsolete: true
Flags: needinfo?(nthomas)
Michelle: We are r? status now and ready for QA. We are on a tight schedule with the launch next week. Here's the QA doc that Ben Niolet created with all of the expected steps and screenshots:

https://docs.google.com/a/mozilla.com/document/d/1GrEluqVH8M0HUM08p7EghKkbS38dc3j3EToFdUvfNkY/edit?usp=sharing
Flags: needinfo?(mfunches)
Thanks Chris & Ben. I am writing the test now, the gdoc makes it a lot faster - Thanks. We should be able to test this tomorrow.
Flags: needinfo?(mfunches)
Attached file all_aboard-1.5.2-fx.xpi (signed) (obsolete) —
:nthomas: latest sign add-on that needs to be spun into the latest v4 of FC 94. Let us know when it is available.
Flags: needinfo?(nthomas)
(In reply to Michelle Funches - QA from comment #22)
> Thanks Chris & Ben. I am writing the test now, the gdoc makes it a lot
> faster - Thanks. We should be able to test this tomorrow.

:michelle: any update on testing?
Flags: needinfo?(mfunches)
Flags: needinfo?(mfunches) → needinfo?(jwilliams)
Hello Chris, our testing can be found here:

https://docs.google.com/spreadsheets/d/1x5_ZvApST5cFfOqg2JePJ6B9GoZPDIPgN3n0OQs-Hd4/edit#gid=640418792

We are still waiting on the email from the "Email Journey" TC (Which should have came in already).
The only issue we found was on win 8.1 & 10 after completing all the sidebar content tracks. On Win 8.1 & 10 the area to put in your info for the sticker is throwing a "Can Not load Page error. We do not encounter this error on Win 7.
Flags: needinfo?(jwilliams)
Sorry the error that is being thrown is "Secure Connection Failed error" not "Can Not load Page error".
Flags: needinfo?(chrismore.bugzilla)
I can speak to the email issue: the journey wasn't turned on until this morning. I would expect any accounts created today should correctly trigger the emails.
Please let me know if you would like a bug to reported on this. I will have the team retest this issue now.
Thank you Ben. We will retest to verify this.
Cool. For your info: the journey cycles every hour on the hour, so any accounts created won't generate the first email until about 5-10 minutes after any given hour.
(In reply to Justin [:JW_SoftvisionQA] from comment #28)
> Please let me know if you would like a bug to reported on this. I will have
> the team retest this issue now.

With regards to the secure connection error. I reckon it might be something up with the SSL cert on Thinksmart's side. We were using a temp URL for the form so, perhaps test with the latest XPI and let us know if the error still happens.

Thanks!
(In reply to Schalk Neethling [:espressive] from comment #31)
> (In reply to Justin [:JW_SoftvisionQA] from comment #28)
> > Please let me know if you would like a bug to reported on this. I will have
> > the team retest this issue now.
> 
> With regards to the secure connection error. I reckon it might be something
> up with the SSL cert on Thinksmart's side. We were using a temp URL for the
> form so, perhaps test with the latest XPI and let us know if the error still
> happens.
> 
> Thanks!

Yeah, the latest XPI above for :nthomas to spin into a new funnelcake should stop this sticker form error.
Flags: needinfo?(chrismore.bugzilla)
The only difference between the funnelcake is testing now vs the add-on in comment 23 is the change to the sticker form URL, which should clear up the issue from above.
Michelle: you can use the exe in comment 34 to re-test the sticker form issue. The rest is the same.
We will get on that.
Win 8.1 and 10 still see the Secured Connection Error with the exe in comment 34. Win 7 works as expected.
Adjusted one sentence below the sticker based off of feedback from legal.
Attachment #8811874 - Attachment is obsolete: true
Attachment #8812822 - Attachment is obsolete: true
(In reply to Justin [:JW_SoftvisionQA] from comment #37)
> Win 8.1 and 10 still see the Secured Connection Error with the exe in
> comment 34. Win 7 works as expected.

Justin: can you confirm you see the same error just by going to https://mozilla.thinksmart.com/ on these same machines?
We found a winner!! The xpi in comment 38 gets rid of the Secured Connection Error on Win 8.1 and 10.
The issue was not on your end it was on ours. We were changing the date ahead to get through each notification and when it got to the sticker page it would not load due to the date being in the future. Sorry about that.
(In reply to Justin [:JW_SoftvisionQA] from comment #41)
> The issue was not on your end it was on ours. We were changing the date
> ahead to get through each notification and when it got to the sticker page
> it would not load due to the date being in the future. Sorry about that.

Oh yes, I should have mentioned that. When you change the clock too far ahead, browsers will throw SSL errors. That is expected. :)

All good then.
:nthomas: once 50.0.1 is ready, please re-spin funnelcake 93 (control vanilla) and funnelcake 94 (onboarding add-on) as we are going to launch today or tomorrow the 23rd (more likely).
Flags: needinfo?(nthomas)
:JW_SoftvisionQA: We are going to go with 50.0 for now and then update to 50.0.1 when it comes out as it could be next week. If you are green, please update this bug asap.
Flags: needinfo?(jwilliams)
Flags: needinfo?(jwilliams)
Funnelcake 50.0 is Green.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
nthomas:

1) build v4 of funnelcake 94 with the XPI in comment 38

2) Upload the builds to the CDN

3) Make bouncer entries for

https://download.mozilla.org/?product=firefox-stub-f93&os=win&lang=en-US (vanilla control)
https://download.mozilla.org/?product=firefox-stub-f94&os=win&lang=en-US (onboarding add-on with migration and xpi)

4) Ping me when read and I will do a full funnel test.

Thank you!
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Did a quick check that the new fc94 repack had the addon installed, firstrun OK, signing OK, queries for updates correctly. Pushed:

s3cmd -c ~/.s3cfg-mozilla cp \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/50.0-candidates/build2/partner-repacks/funnelcake93/win32/en-US/Firefox Setup 50.0.exe" \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/releases/50.0-funnelcake93/win32/en-US/Firefox Setup 50.0.exe"

s3cmd -c ~/.s3cfg-mozilla cp \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/50.0-candidates/build2/partner-repacks/funnelcake93/win32/en-US/Firefox Setup 50.0.exe.asc" \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/releases/50.0-funnelcake93/win32/en-US/Firefox Setup 50.0.exe.asc"

s3cmd -c ~/.s3cfg-mozilla cp \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/50.0-candidates/build2/partner-repacks/funnelcake94/win32/en-US/Firefox Setup 50.0.exe" \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/releases/50.0-funnelcake94/win32/en-US/Firefox Setup 50.0.exe"

s3cmd -c ~/.s3cfg-mozilla cp \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/candidates/50.0-candidates/build2/partner-repacks/funnelcake94/win32/en-US/Firefox Setup 50.0.exe.asc" \
"s3://net-mozaws-prod-delivery-firefox/pub/firefox/releases/50.0-funnelcake94/win32/en-US/Firefox Setup 50.0.exe.asc"


Bouncer setup, links are live at:
https://download.mozilla.org/?product=firefox-stub-f93&os=win&lang=en-US (vanilla control)
https://download.mozilla.org/?product=firefox-stub-f94&os=win&lang=en-US (onboarding add-on with migration and xpi)

Verified SHA1 match comment 10/20 and 47.

Leaving open for the 50.0.1 respin.
Tested the full funnel on my Windows machine and all looks good. We will launch tomorrow. Thanks
Summary: Funnelcakes 93/94 50.0.1 for Onboarding Add-on Experiment (11/22/2016 launch) → Funnelcakes 93/94 50.0.1 for Onboarding Add-on Experiment (11/23/2016 launch)
Depends on: 1319880
:nthomas: Here's the approved and signed XPI from the AMO team. This should be the XPI used for the final funnelcake. Sorry for the last re-spin of the funnelcake. Thank you.
Attachment #8812946 - Attachment is obsolete: true
Flags: needinfo?(nthomas)
Config updated at https://github.com/mozilla-partners/funnelcake/commit/9caf64aa61c7866a9dc95778003d7b1028748f06

fc94:
https://archive.mozilla.org/pub/firefox/candidates/50.0-candidates/build2/partner-repacks/funnelcake94/win32/en-US/Firefox%20Setup%2050.0.exe (SHA1: b232b219311e06e7734f792570b4c871852271fd)

NB: overwrote build at comment #47, but CDN caching may serve the previous file - check the SHA1! If you have mac/linux this is 'openssl dgst -sha1 <file>' or 'sha1sum <file>'.

Also pushed to firefox/releases/50.0-funnelcake94/. Bug 1319880 should have purged this location, but still check the SHA1. I'm still getting 34f9acedf3c8b5fcd116b385bb3035b20c72b996 so I'll take that up with CloudOps. Please hold off enabling for now.
Flags: needinfo?(nthomas)
I'm getting b232b219311e06e7734f792570b4c871852271fd now that oremj has sorted Akamai out. Green light to enable distribution.
Refreshed funnelcake builds based on 50.0.1 build1, no config changes.

funnelcake93:
https://archive.mozilla.org/pub/firefox/candidates/50.0.1-candidates/build1/partner-repacks/funnelcake93/win32/en-US/Firefox%20Setup%2050.0.1.exe (SHA1: 4249768a861424304c55a84e1ec8a1118cf04072)

funnelcake95:
https://archive.mozilla.org/pub/firefox/candidates/50.0.1-candidates/build1/partner-repacks/funnelcake94/win32/en-US/Firefox%20Setup%2050.0.1.exe (SHA1: d63edf328b4c3577300f84ee32e768f8f1b498bd)
(In reply to Nick Thomas [:nthomas] from comment #53)
> Refreshed funnelcake builds based on 50.0.1 build1, no config changes.
> 
> funnelcake93:
> https://archive.mozilla.org/pub/firefox/candidates/50.0.1-candidates/build1/
> partner-repacks/funnelcake93/win32/en-US/Firefox%20Setup%2050.0.1.exe (SHA1:
> 4249768a861424304c55a84e1ec8a1118cf04072)
> 
> funnelcake95:
> https://archive.mozilla.org/pub/firefox/candidates/50.0.1-candidates/build1/
> partner-repacks/funnelcake94/win32/en-US/Firefox%20Setup%2050.0.1.exe (SHA1:
> d63edf328b4c3577300f84ee32e768f8f1b498bd)

Do you mean 93 and 94?

and you'll push out 50.0.1 to the CD and update bouncer after 50.0.1 is out the released?
Sorry, that should be 'funnelcake94:'. I'm happy to push to the CDN and update bouncer after the main release goes out, let me know how that fits with the US holidays though.
Funnelcakes started at 7:05pm pst on 11/25/2016.
Updated bouncer to point to the 50.0.1 files from comment #54.

To avoid a ~5 minute period of 404s for end users I added locations for the 50.0.1 files with faked OS of osxppc, let sentry find them, swapped the OS=win locations from 50.0 to 50.0.1, then removed the osxppc locations.


BTW, I saw this in the browser console with fc94:
all-aboard-v1-5:Either no config.json file was created, or it was placed at the wrong location. Error: Exception { message: "Component returned failure code: 0x…", result: 2152857618, name: "NS_ERROR_FILE_NOT_FOUND", filename: "resource://gre/modules/commonjs/too…", lineNumber: 84, columnNumber: 0, data: null, stack: "readURISync@resource://gre/modules/…", location: XPCWrappedNative_NoHelper }  utils.js:34

It's also present in the 50.0 version of fc94, so is likely to be cosmetic.
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
(In reply to Nick Thomas [:nthomas] from comment #57)
> Updated bouncer to point to the 50.0.1 files from comment #54.
> 
> To avoid a ~5 minute period of 404s for end users I added locations for the
> 50.0.1 files with faked OS of osxppc, let sentry find them, swapped the
> OS=win locations from 50.0 to 50.0.1, then removed the osxppc locations.
> 
> 
> BTW, I saw this in the browser console with fc94:
> all-aboard-v1-5:Either no config.json file was created, or it was placed at
> the wrong location. Error: Exception { message: "Component returned failure
> code: 0x…", result: 2152857618, name: "NS_ERROR_FILE_NOT_FOUND", filename:
> "resource://gre/modules/commonjs/too…", lineNumber: 84, columnNumber: 0,
> data: null, stack: "readURISync@resource://gre/modules/…", location:
> XPCWrappedNative_NoHelper }  utils.js:34

That error is normal. When we are doing local testing, we put a config.json in the root folder of the XPI, it and allows us to override the default variables to make testing easier. Then we remove that in the production version.

> 
> It's also present in the 50.0 version of fc94, so is likely to be cosmetic.

Thanks!
Reopen to generate 50.0.2 based builds.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Did super-quick QA and the correct firstrun pages came up, with modification on fc94; pushed to firefox/releases/; updated bouncer to point to 50.0.2; verified SHA1 are correct.
cmore, did we disable distribution for this and bug 1315783 yet ? I ask because we shipped vanilla 50.1.0 on Tuesday PT.

50.1.0 based builds are available in https://archive.mozilla.org/pub/firefox/candidates/50.1.0-candidates/build2/partner-repacks/ because I was working on bug 1323022. Not available via bouncer yet though.
Yup, I was just out to update this bug.

Distribution stopped automatically on 12/12/2016. All good here. Results should be back in at the end of December or first week in January.
Nice.
Status: REOPENED → RESOLVED
Closed: 7 years ago7 years ago
Resolution: --- → FIXED
Component: Custom Release Requests → Release Requests
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: