Funnelcakes to test impact of new first and second run pages for Firefox 46.0

RESOLVED FIXED

Status

Release Engineering
Releases: Custom Builds
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: cmore, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

2 years ago
We need to start a funnelcake test to test the engagement and retention impact of new first run and second run pages.

We need 7 separate funnelcake (funnelcake 75 through 81) 

Language: en-US
Version: Firefox 46.0 (or newer if we chemspill this week)
OS: Windows

For reference, each of the 7 variations test the following changes:

v1 (control):  Re-themed Accounts First Run / No secondrun
v2: Re-themed Accounts First Run / Private browsing + mobile buttons second run
v3: Re-themed Accounts First Run / pinning Firefox + searching second run
v4: Private browsing + mobile buttons first run/Re-themed Accounts Second Run
v5: Pinning Firefox + searching firstrun / complex accounts second run
v6: Complex accounts sign up firstrun / Pinning Firfefox + searching second run 
v7: No first run or second run

This test will be testing new Firefox functionality for later run pages created with bug 1200639 with logic like:

pref("browser.laterrun.pages.test.url", "http://www.mozilla.org/");
pref("browser.laterrun.pages.test.minTime", 24);
pref("browser.laterrun.pages.test.minSessionCount", 3);
pref("browser.laterrun.pages.test.requireBoth", true);

Here's an example funnelcake config for funnelcake 76 v2 variation:

--------

distribution.ini

[Global]
id=mozilla76
version=1.0
about=Funnelcake Q2 2016 - First and Second Run pages

[Preferences]
app.partner.mozilla76="mozilla76"

[LocalizablePreferences]
startup.homepage_welcome_url="https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/firstrun/?f=76&v=2"

startup.homepage_override_url="https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/whatsnew/?oldversion=%OLD_VERSION%&f=76"

browser.laterrun.pages.test.url="https://www.mozilla.org/%LOCALE%/%APP%/%VERSION%/secondrun/?f=76&v=2"

browser.laterrun.pages.test.minTime="24"
browser.laterrun.pages.test.minSessionCount="3"
browser.laterrun.pages.test.requireBoth="true"

repack.cfg:

aus="funnelcake76"
dist_id="funnelcake76"
dist_version="1.0"
linux-i686=false
locales="en-US"
mac=false
win32=true

# Upload params
s3cfg=~/.s3cfg-mozilla
bucket="net-mozaws-prod-delivery-firefox"
upload_to_candidates=true

----------

I will reply with additional comments for the funnelcake configs for each of the 7 funnelcakes. They should follow a similar format to the example above.
(Reporter)

Comment 1

2 years ago
verdi/jpetto: can you review the example config for funnelcake 76 with the v2 variation above and see if it looks right? If so, I will duplicate it for each of the 7 funnelcakes for the releng team doesn't have to guess.
Flags: needinfo?(mverdi)
Flags: needinfo?(jon)
(Reporter)

Updated

2 years ago
Depends on: 1267308
(Reporter)

Updated

2 years ago
Blocks: 1267308
No longer depends on: 1267308
(Reporter)

Updated

2 years ago
Blocks: 1259608

Comment 2

2 years ago
My first time looking at a funnelcake config, but I don't see any obvious errors or typos.

Just to clarify, minTime refers to hours, correct? Second run will be shown only after 24 hours *and* 3 sessions?
Flags: needinfo?(jon)

Comment 3

2 years ago
Also my first time looking at a funnelcake config. The only things I notice that don't look as I expected are:
1. I thought the pages (specifically the pin Firefox page) included a version for Mac so we could run this for Mac users.
2. We should not be showing this to XP users. The pin Firefox instructions don't work for them.
Flags: needinfo?(mverdi)
(Reporter)

Comment 4

2 years ago
Will note that for this experiment, we'll *not* target XP. Funnelcakes are only one OS and to do a Mac + Windows, it doubled the amount of funnelcakes. For now, we'll do Windows only and can do follow up smaller one for specific OS after.
(Reporter)

Comment 5

2 years ago
:coop :nthomas: we need to get these builds together. What do you need to move forward?

Check out comment 0. Let me know if you need me to write out the specific configs for each funnelcake. If so, do you want me to do it as a separate comment, Google Doc, or what? This could get confusing with all of the variations.
Flags: needinfo?(nthomas)
Flags: needinfo?(coop)
(Assignee)

Comment 6

2 years ago
(In reply to Chris More [:cmore] from comment #5)
> Check out comment 0. Let me know if you need me to write out the specific
> configs for each funnelcake. If so, do you want me to do it as a separate
> comment, Google Doc, or what? This could get confusing with all of the
> variations.

The configs are all in github now, so you can just submit the changes as a PR:

https://github.com/mozilla-partners/funnelcake
Flags: needinfo?(nthomas)
Flags: needinfo?(coop)
(Reporter)

Comment 7

2 years ago
(In reply to Chris Cooper [:coop] from comment #6)
> (In reply to Chris More [:cmore] from comment #5)
> > Check out comment 0. Let me know if you need me to write out the specific
> > configs for each funnelcake. If so, do you want me to do it as a separate
> > comment, Google Doc, or what? This could get confusing with all of the
> > variations.
> 
> The configs are all in github now, so you can just submit the changes as a
> PR:
> 
> https://github.com/mozilla-partners/funnelcake

Can you create the configs and post them as a PR and I can r? on them? I have zero bandwidth right now to create the configs and submit anything. We need to get the funnelcake out asap or this whole project and all the work that was done will be dead in the water.

Would it be more helpful if I spell out the specific changes per variation?

Thanks
Flags: needinfo?(coop)
(Assignee)

Comment 8

2 years ago
(In reply to Chris More [:cmore] from comment #7)
> Can you create the configs and post them as a PR and I can r? on them? I
> have zero bandwidth right now to create the configs and submit anything. We
> need to get the funnelcake out asap or this whole project and all the work
> that was done will be dead in the water.

Chris: I appreciate that you're busy. So are we. We are especially busy during release weeks, i.e. when this request was initiated.

I also appreciate that some non-trivial amount of work has been spent to organize this funnelcake. While it would be sad to lose that work, it's exceedingly hard to prioritize work across teams, especially when that work arrives unannounced with a quick deadline.

Releng is a necessary hurdle for you right now in terms of creating signed builds and getting funnelcake information into bouncer. We have no desire to add another releng hurdle to your process.

In general, releng has no knowledge about what you're trying to accomplish in a given funnelcake. The effort spent describing the funnelcake, e.g. comment #0, would be better spent just writing the configs.

In the future, part of the process for putting together a funnelcake campaign *needs* to be creating the configs for that campaign.

Releng will make a best-effort to get these particular configs ready for you in short order, but this is the last time we will do so.
Flags: needinfo?(coop)
Created attachment 8748041 [details] [review]
Pull Request
Attachment #8748041 - Flags: review?(chrismore.bugzilla)
(Reporter)

Comment 10

2 years ago
(In reply to Nick Thomas [:nthomas] from comment #9)
> Created attachment 8748041 [details] [review]
> Pull Request

Awesome job!

I couldn't find a single problem with your configs.

I have asked jpetto for another set of eyes on it so we don't miss something.
(Reporter)

Comment 11

2 years ago
Comment on attachment 8748041 [details] [review]
Pull Request

Looks good. Asking :jpetto for a r? too.
Attachment #8748041 - Flags: review?(chrismore.bugzilla) → review+
(Reporter)

Comment 12

2 years ago
Two comments (question further down):

1) These builds will have to be created with 46.0.1 since that is the latest version out.

2) The d.m.o entries for these builds will have to look like stub products even though they will be full installers so that mozilla.org can distribute them without having to create one-off JS code. For example:

https://download.mozilla.org/?product=firefox-stub-f75&os=win&lang=en-US 
https://download.mozilla.org/?product=firefox-stub-f76&os=win&lang=en-US 
...
https://download.mozilla.org/?product=firefox-stub-f81&os=win&lang=en-US 

Ask: How long will it take to get the builds ready to test?

Given the sample size and number of variations, everything will need to be tested, pushed to production, and the experiment launched by May 11th. 

Thanks in advance for everything!
I'll get them generated so someone can do some verification.
After much debugging the builds are up at:
 http://archive.mozilla.org/pub/firefox/candidates/46.0.1-candidates/build1/funnelcake75/win32/en-US/Firefox%20Setup%2046.0.1.exe
 http://archive.mozilla.org/pub/firefox/candidates/46.0.1-candidates/build1/funnelcake76/win32/en-US/Firefox%20Setup%2046.0.1.exe
 http://archive.mozilla.org/pub/firefox/candidates/46.0.1-candidates/build1/funnelcake77/win32/en-US/Firefox%20Setup%2046.0.1.exe
 http://archive.mozilla.org/pub/firefox/candidates/46.0.1-candidates/build1/funnelcake78/win32/en-US/Firefox%20Setup%2046.0.1.exe
 http://archive.mozilla.org/pub/firefox/candidates/46.0.1-candidates/build1/funnelcake79/win32/en-US/Firefox%20Setup%2046.0.1.exe
 http://archive.mozilla.org/pub/firefox/candidates/46.0.1-candidates/build1/funnelcake80/win32/en-US/Firefox%20Setup%2046.0.1.exe
 http://archive.mozilla.org/pub/firefox/candidates/46.0.1-candidates/build1/funnelcake81/win32/en-US/Firefox%20Setup%2046.0.1.exe

An additional pull request was needed (https://github.com/mozilla-partners/funnelcake/pull/17), and we ran into bug 1270707 but could hardcode around it.

cmore, could you please organize someone to test these builds ? It's much easier to fix them before pushing into production locations.
Flags: needinfo?(chrismore.bugzilla)
Bah, the installers aren't signed, I'll investigate.
coop, this is how I was generating the builds, iterating over the --partner arg as there are unneeded funnelcakes in desktop which I didn't know the state of:

partner-repack1:scripts cltbld$ ./partner-repacks.py -v ${VERSION}    -n ${BUILDNUMBER}    -d ../partners/funnelcake/desktop --partner=funnelcake75
2016-05-05 20:33:28,429 - INFO - ##################################################
2016-05-05 20:33:28,430 - DEBUG - Found partner config: ../partners/funnelcake/desktop/funnelcake75 ['distribution'] repack.cfg
2016-05-05 20:33:28,434 - INFO - Creating 4 Repack Workers
2016-05-05 20:33:28,437 - INFO - Starting repack process for partner: funnelcake75
2016-05-05 20:33:28,442 - INFO - Found /Users/cltbld/funnelcake/scripts/original_builds/46.0.1/build1/win32/en-US/Firefox Setup 46.0.1.exe on disk, not downloading
2016-05-05 20:33:28,443 - INFO - Repacking win32 build Firefox Setup 46.0.1.exe
2016-05-05 20:33:29,150 - DEBUG - Executing 7za a "Firefox Setup 46.0.1.exe" core
2016-05-05 20:33:29,150 - DEBUG - in /Users/cltbld/funnelcake/scripts/repacked_builds/signed/funnelcake75/46.0.1-1/funnelcake75/win32/en-US/working

7-Zip (A) [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=utf8,Utf16=on,HugeFiles=on,4 CPUs)

Scanning

Updating archive Firefox Setup 46.0.1.exe

Compressing  core/distribution/distribution.ini

Everything is Ok
2016-05-05 20:33:29,675 - DEBUG - Executing python /Users/cltbld/funnelcake/scripts/tools/release/signing/signtool.py -t /Users/cltbld/funnelcake/scripts/token -n /Users/cltbld/funnelcake/scripts/nonce -c /Users/cltbld/funnelcake/scripts/tools/release/signing/host.cert -H gpg:osslsigncode:mar:jar:b2gmar:emevoucher:sha2signcode:signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 --formats sha2signcode --formats gpg "Firefox Setup 46.0.1.exe"
2016-05-05 20:33:29,676 - DEBUG - in /Users/cltbld/funnelcake/scripts/repacked_builds/signed/funnelcake75/46.0.1-1/funnelcake75/win32/en-US/working
2016-05-05 20:33:30,090 - Skipping Firefox Setup 46.0.1.exe because it looks like it's already signed  <<<<------- OMGWTHBBQ
2016-05-05 20:33:30,247 - 672bb519499a4885f902fdd399aa278aa86141c3: processing Firefox Setup 46.0.1.exe on https://signing4.srv.releng.scl3.mozilla.com:9120
2016-05-05 20:33:30,262 - 672bb519499a4885f902fdd399aa278aa86141c3: uploading for signing
2016-05-05 20:33:33,775 - 672bb519499a4885f902fdd399aa278aa86141c3: processing Firefox Setup 46.0.1.exe on https://signing4.srv.releng.scl3.mozilla.com:9120
2016-05-05 20:33:33,788 - 672bb519499a4885f902fdd399aa278aa86141c3: OK
2016-05-05 20:33:33,802 - INFO - Done repacking win32 build Firefox Setup 46.0.1.exe

The problem looks to be the line with the (subtle) marker, but this doesn't match my mental model for how these repacks are down (tear the installer apart, modify the bundle of files in core, put it together again). We also don't seem to have this problem for 46.0.1 EME builds. My method for checking signing is just to look at the Properties of the installer in Windows Explorer, where there's no Digital Signature Tab for the builds in comment #14.

cmore, clearing your ni? for now, because the builds aren't completely ready. We could still check the funnelcake functionality though, or nominate your QA person.
Flags: needinfo?(chrismore.bugzilla) → needinfo?(coop)
(Assignee)

Comment 17

2 years ago
(In reply to Nick Thomas [:nthomas] from comment #16)
> The problem looks to be the line with the (subtle) marker, but this doesn't
> match my mental model for how these repacks are down (tear the installer
> apart, modify the bundle of files in core, put it together again). We also
> don't seem to have this problem for 46.0.1 EME builds. My method for
> checking signing is just to look at the Properties of the installer in
> Windows Explorer, where there's no Digital Signature Tab for the builds in
> comment #14.

I haven't seen this before, but can confirm it after trying the repack process myself.

I have a little time today to investigate to try to get this unblocked.
Assignee: nobody → coop
Status: NEW → ASSIGNED
Flags: needinfo?(coop)
(Reporter)

Comment 18

2 years ago
Thanks for the update and I appreciate the help. Let me know when they are ready to test.
(Assignee)

Comment 19

2 years ago
EME-free repacks appears to be affected now too:

016-05-07 19:35:50,379 - DEBUG - Executing python /Users/cltbld/mozilla-EME-free/scripts/tools/release/signing/signtool.py -t /Users/cltbld/mozilla-EME-free/scripts/token -n /Users/cltbld/mozilla-EME-free/scripts/nonce -c /Users/cltbld/mozilla-EME-free/scripts/tools/release/signing/host.cert -H gpg:osslsigncode:mar:jar:b2gmar:emevoucher:sha2signcode:signing4.srv.releng.scl3.mozilla.com:9120 -H dmgv2:mac-v2-signing1.srv.releng.scl3.mozilla.com:9120 --formats sha2signcode --formats gpg "Firefox Setup 46.0.1.exe"
2016-05-07 19:35:50,379 - DEBUG - in /Users/cltbld/mozilla-EME-free/scripts/repacked_builds/signed/win32-EME-free/ach/working
2016-05-07 19:35:50,851 - Skipping Firefox Setup 46.0.1.exe because it looks like it's already signed
2016-05-07 19:35:51,008 - e2e1851210c3e6c801b74c5c0b51679160b32a6b: processing Firefox Setup 46.0.1.exe on https://signing4.srv.releng.scl3.mozilla.com:9120
2016-05-07 19:35:51,022 - e2e1851210c3e6c801b74c5c0b51679160b32a6b: uploading for signing

It uploads the file for gpg signing, but not for Windows signing.

That particular error message is coming from is_authenticode_signed() in signtool.py. That code hasn't changed recently, but perhaps the vars it's checking for have?

https://hg.mozilla.org/build/tools/file/b030af490d93/release/signing/signtool.py#l23

I confess that I'm out of my depth with the signing. Pulling in Ben to help.
Flags: needinfo?(bhearsum)
Thanks for looking into it coop. I've traced back the problem to the way I ran the repack scripts the first time, which was without the --partner <foo> argument. The script detected funnelcake 63-68 were present and I panic-cancelled it to avoid them, but happened to do so right in the middle of the call to repack-signed.sh. Consequently we had an
  original_builds/46.0.1/build1/win32/en-US/Firefox Setup 46.0.1.exe
which was the vanilla build, and hadn't been replaced by an unsigned exe after decompress & recompress.

I've now purged original_builds/46.0.1/ and rerun the script, which yielded properly signed installers.
Flags: needinfo?(bhearsum)
cmore, lets try this again. Please organize QA for the builds linked in comment #14. If possible verify the file was modified May 9 instead of older (since we're replacing files on a system that caches).
Flags: needinfo?(chrismore.bugzilla)
(Reporter)

Comment 22

2 years ago
Will do! Thanks
Flags: needinfo?(chrismore.bugzilla)
(Reporter)

Comment 23

2 years ago
:ryanVM: would you or your team have any availability the next few days to help me QA test these builds? I can put together a quick test plan. Let me know if you want some more context. Thanks
Flags: needinfo?(ryanvm)
(Reporter)

Comment 24

2 years ago
I am doing some basic functional testing now and putting in the results of it in this test plan doc:

https://docs.google.com/document/d/1EPKZr-Cw_8zNbKae68eCeKhkfAx7YKwvkNiF0jAylWg/edit?usp=sharing
What sort of timeframe are we on here? Judging by the comments in the bug, I gather that this is semi-urgent?  What sort of testing are you looking for?
Flags: needinfo?(ryanvm) → needinfo?(chrismore.bugzilla)
(Reporter)

Comment 26

2 years ago
(In reply to Ryan VanderMeulen [:RyanVM] from comment #25)
> What sort of timeframe are we on here? Judging by the comments in the bug, I
> gather that this is semi-urgent?  What sort of testing are you looking for?

I dropped in the basic things I want to test for each build in the doc in comment 24. 

For now, I just want to make sure each builds installs and what exact URL opens up on first run. 

Timing: I am trying to get the funnelcakes released by this Friday as I am running out of time to get enough data before Firefox 47.0 is released in June. 

I am working on doing some functional testing too and any help is appreciated.
Flags: needinfo?(chrismore.bugzilla) → needinfo?(ryanvm)
I'll have the folks in Las Vegas take a look.
Flags: needinfo?(ryanvm) → needinfo?(mfunches)
Acknowledge: Vegas can assist with this.
There are 5 of us here we will apply 77-81.
You can contact me on irc Michelle-QA
Flags: needinfo?(mfunches)
(Reporter)

Comment 29

2 years ago
:nthomas: the QA of the builds is coming along well. Everything is looking good thus far. I don't foresee any issues and timing is getting critical with getting this out the door in time.

What do we need to do to get the bouncer entries set up as defined in comment 12?

As mentioned in comment 12, we are going to pretend these are stub builds and they stub product will just point to these full installers. This will eliminate from having to do any extra JS on www.mozilla.org to distribute them.
Flags: needinfo?(nthomas)
Bouncer is set up, let me know when you want to copy the builds to the release location (eg firefox/releases/46.0.1-funnelcake75).
Flags: needinfo?(nthomas)
Builds are copied to final location, eg http://archive.mozilla.org/pub/firefox/releases/46.0.1-funnelcake75/win32/en-US/, to enable end-to-end testing.

If we end up with issues found by QA and need to regenerate the builds then I strongly suggest we ask CloudOps to purge CDNs the at prefixes
  http://archive.mozilla.org/pub/firefox/releases/46.0.1-funnelcake75/
  http://archive.mozilla.org/pub/firefox/releases/46.0.1-funnelcake75/
etc etc. That's CloudFront for our product delivery system, and then all the CDNs used in download.cdn.mozilla.net (Akamai etc). oremj & jthomas are good people to talk to.
(Reporter)

Comment 32

2 years ago
I've done another full funnel test and bouncer looks to be setup perfectly. All the QA work from softvision looked great and it is all as expected. I feel good about good about getting this retention test started now.
(Reporter)

Comment 33

2 years ago
Thanks everyone!
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Reporter)

Updated

a year ago
Blocks: 1284627
(Reporter)

Updated

a year ago
No longer blocks: 1284627
You need to log in before you can comment on or make changes to this bug.