1191472 - Create funnelcake builds for Firefox 40 Windows 10 marketing campaign

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Chris More [:cmore]]

Build:

* en-US
* win32
* version: Firefox 40.0 release

We are going to use this to measure MAUs and funnel analysis for a Windows campaign that Eric/Winston/Fabio are running. 

All I need is a single funnelcake build created, no changes, just the channel name. I think the next one release-cck-mozilla42 based off of my tracker, but I think Shane may have spun one up too for another need. Just let me know the next funnelcake ID and I will get it logged.

Let me know when the build is ready and when the URL is available on download.mozilla.org. Campaign is going to be released around August 13th.

Thanks!

Comment 1

[Chris More [:cmore]]

:nthomas: given that fx40 is next week, what's the timeline for spinning up a funnelcake to do some cohort analysis for Eric's campaign?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Somewhat complicated by the number of 40.0 RCs, the late request, and who-has-precedence with bug 1191060. Do you want to ship this on Tuesday when 40.0 goes live ?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Oh, you said Aug 13th (reading is good). That will leave use time wait for the RCs to stabilise.

Please comment on
* how this sits compared to bug 1191060, and which funsize number this should be
* if you'd like the usual firstrun and whatsnew pages

Comment 4

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #3)
> Oh, you said Aug 13th (reading is good). That will leave use time wait for
> the RCs to stabilise.
> 
> Please comment on
> * how this sits compared to bug 1191060, and which funsize number this
> should be
> * if you'd like the usual firstrun and whatsnew pages

Good questions. Seems like everyone wants to do funnelcakes now! I think I may have told too many people what is possible with funnelcakes and it is now a buzzword. :)

> * how this sits compared to bug 1191060, and which funsize number this should be

Hmmm, well the funnelcake for default browser is pretty important, but the funenlcake is also important to understand ROI of Eric's campaign. So, I would say it is equal priority.

The updated go live date for this funnelcake Monday, August 17th. Does that help?

> * if you'd like the usual firstrun and whatsnew pages

Yes, just a vanilla funnelcake. Nothing special, just ?f=XX on firstrun/whatsnew. I'll control the rest with deployment of it.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Sounds like the default browser test isn't ready to go, based on bug 1165372 comment #8. If that changes, can we run tests at the same time ?

Allocated funnelcake44 for this bug, and assumed Distribution Point and Distribution by would be firefox/new and Optimizely.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

We're up to 40.0.2 already. I'll do a funnelcake with that but I'm not 100% confident we're done yet.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

cmore, the build is up for your verification at 
https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.2-candidates/build1/partner-repacks-bug1191472/funnelcake44/win32/en-US/Firefox%20Setup%2040.0.2.exe

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

cmore, what's our status here ?

Comment 9

[Chris More [:cmore]]

nthomas: sorry, got pulled into lot of requests last week. 40.0.2 is perfect. 

Can you make a bouncer entry at:

https://download.mozilla.org/?product=firefox-40.0.2-SSL-f44&os=win&lang=en-US

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

It's working now.

Cleaned up config - https://hg.mozilla.org/build/partner-repacks/rev/f11452febb61

Comment 11

[Chris More [:cmore]]

Experiment set up at:

https://app.optimizely.com/edit?experiment_id=3397430197

URL targeting:

^.*www\.mozilla\.org\/en\-US\/firefox\/new\/.+utm_campaign=(fx\-display\-2015).+utm_content=(pr|pe).*$

Audience: Non-Firefox, Desktop, English, Windows only

Variations: FC 44 variation

Goal: Click through rate on download button (not really the purpose of the funnelcake, but you have to use some goal for a test to distribute something)

Code for FC 44 variation: $(".os_win > .download-link").attr({"href":"https://download.mozilla.org/?product=firefox-40.0.2-SSL-f44&os=win&lang=en-US"});

Traffic allocation: 100% matching URL targeting and Audience conditions.

Comment 12

[Chris More [:cmore]]

Example URL that we need to distribute the funnelcake to:

https://www.mozilla.org/en-US/firefox/new/?utm_source=jmpbp&utm_medium=display&utm_campaign=fx-display-2015&utm_content=pe-slow-300x250

Comment 13

[Chris More [:cmore]]

:jpetto: can you review the experiment above on comment 11? It's a duplicate of the one that Cory setup for the Windows 10 ad-campaign, but with only one variation and it is to distribute the funnelcake by changing the URL to the download.

Comment 14

[Chris More [:cmore]]

:verdi: can you use the exact link in comment 9 and install it on a windows machine (any version) and let me know the URL of the /firstrun/? It should contain an ?f=44

Comment 15

[Verdi [:verdi] Best to slack me]

It worked. My first run url was https://www.mozilla.org/en-US/firefox/40.0.2/firstrun/?f=44

Comment 16

[Jonathan Petto [:jpetto]]

Little hesitant to hard code a download link (even in a test), but code looks good and seems to function properly. Tested on Win 8.1/IE 11 VM.

Just need to make sure this test doesn't run past 40.0.2.

Comment 17

[Chris More [:cmore]]

(In reply to Jon Petto [:jpetto] from comment #16)
> Little hesitant to hard code a download link (even in a test), but code
> looks good and seems to function properly. Tested on Win 8.1/IE 11 VM.
> 
> Just need to make sure this test doesn't run past 40.0.2.

For sure. All 20+ funnelcakes we have done the past few years have been done exactly like this and I share the same concern, but it about the only we can on an ad hoc basis serve up a different build to a discrete audience for only a certain amount of days with our current setup. Agree that this test can't run past 9/22/2015 or if there is any build after 40.0.2 created.

Thanks, jpetto!

Comment 18

[Chris More [:cmore]]

Attachment: Added optimizely schedule to shut off test on 9/21 before Fx 41

The test will be shut off before 9/21, but the backup is that the test will be scheduled to shut itself off on that date if no one does manually.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

40.0.3 is coming, would you like a funnelcake44 for that ?

Comment 20

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #19)
> 40.0.3 is coming, would you like a funnelcake44 for that ?

Yikes, another dot release. What's the ETA on shipping 40.0.3? Yes, please spin up a 40.0.3 funnelcake44 with the same everything. Would the D.M.O. link stay the same except for the version number?

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

Shipping in the next 24 hours, likely daytime Pacific. Usual disclaimer about that slipping if snags or something else comes up, so check with the usual suspects.

Funnelcake build is up for verification - https://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1191472/funnelcake44/win32/en-US/Firefox%20Setup%2040.0.3.exe

Bouncer link will be 
  https://download.mozilla.org/?product=firefox-40.0.3-SSL-f44&os=win&lang=en-US
(ie only change 40.0.2 -> 40.0.3)

RelEng: just need to do this once cmore says build is OK, and we've pushed the vanilla builds to 'mirrors':
# ffxbld@stage
rsync -av /pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1191472/funnelcake44/ /pub/mozilla.org/firefox/releases/40.0.3-funnelcake44/

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Ran the rsync, and verified the download.m.o link works.

Please sanity check the build before enabling the experiment.

Comment 23

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #22)
> Ran the rsync, and verified the download.m.o link works.
> 
> Please sanity check the build before enabling the experiment.

Thanks, Nick!

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Distribution finished during September.

Comment 25

[Chris More [:cmore]]

FYI. The results of this funnelcake were lost because the distribution of the funnelcake was during the 30 days FHR was not collecting records due to bug 1209664. Given that the funnelcake was a retention test, any profile that was born and died during that time period is lost. This means we cannot get conclusive data from this test.