1184279 - Create funnelcake builds to test Firefox homepage variations

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Francesco Polizzi [:fpolizzi]]

Background:
According to data from FHR and Testpilot, only a small percentage of all users have about:home set as their default homepage. Although browser hijacking is an issue, a large percentage of these users are choosing other experiences over about:home intentionally.

The purpose of this test is to measure these changes with a change in the default homepage:
1. Impact on user's search volume
2. Impact on retention of the user

Both of these data points we can gather and segment by funnelcake via FHR.

Requirements:
1. Funnelcake build which has about:home set as the homepage (control)
2. Funnelcake build which has about:newtab set as the homepage (variation)

This project is time sensitive, and would be great to get in as soon as possible.

Comment 1

[Shane Caraveo (:mixedpuppy)]

Attachment: funnelcake_homepage

my first funnelcakes...

Comment 2

[Shane Caraveo (:mixedpuppy)]

Attachment: funnelcake_homepage

Comment 3

[Shane Caraveo (:mixedpuppy)]

https://hg.mozilla.org/build/partner-repacks/rev/87c71751ca71

@rail, can you spin builds for that?

Comment 4

[Chris More [:cmore]]

I usually work with :nthomas on funnelcake creations and he usually adds them as a product to download.mozilla.org (DMO). Nick can probably review this for us. 

Nick?

Comment 5

[Chris More [:cmore]]

Also, we have already used funnelcake 39 IDs, so this will have to be funnelcake 40 and 41.

Funnelcake master tracker:

https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0

Comment 6

[Chris More [:cmore]]

Looks like 39 was changed to 40, so we should be fine. We need two funnelcakes (40 and 41) so we can do an A/B test and have a control.

Comment 7

[Shane Caraveo (:mixedpuppy)]

(In reply to Chris More [:cmore] from comment #6)
> Looks like 39 was changed to 40, so we should be fine. We need two
> funnelcakes (40 and 41) so we can do an A/B test and have a control.

Ok, you probably want the initial patch I did in comment #1

Comment 8

[Chris More [:cmore]]

(In reply to Shane Caraveo (:mixedpuppy) from comment #7)
> (In reply to Chris More [:cmore] from comment #6)
> > Looks like 39 was changed to 40, so we should be fine. We need two
> > funnelcakes (40 and 41) so we can do an A/B test and have a control.
> 
> Ok, you probably want the initial patch I did in comment #1

Nice, Shane! Quick turn around. :)

Comment 9

[Chris More [:cmore]]

For the funnelcakes:

mozilla40: about:home (control and no changes)
mozilla41: about:newtab (variation with about:newtab set as homepage)

Language: en-US

Platform: Win32

This looks like what Shane has set up.

Comment 10

[Shane Caraveo (:mixedpuppy)]

I'll need to revert my last push and push the older patch.

Comment 11

[Shane Caraveo (:mixedpuppy)]

Just pushed a patch to get funnelcake41 right.

https://hg.mozilla.org/build/partner-repacks/rev/708cc3099338

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

I can make some repacks once we've settled in the repo. Curious why we're setting startup.homepage_override_url ? That's the post-update page.

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

That pref isn't set in the vanilla builds, so no page will be shown after an update (unless the update server overrides that). I'd recommend leaving it alone unless you have something end in mind.

Comment 14

[Shane Caraveo (:mixedpuppy)]

(In reply to Nick Thomas [:nthomas] from comment #13)
> That pref isn't set in the vanilla builds, so no page will be shown after an
> update (unless the update server overrides that). I'd recommend leaving it
> alone unless you have something end in mind.

No, I just thought they both had to be set.  I'll remove that line in both.

Comment 15

[Shane Caraveo (:mixedpuppy)]

There ya go

https://hg.mozilla.org/build/partner-repacks/rev/116a473d68be

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Builds up for testing at 
http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/39.0-candidates/build6/partner-repacks-bug1184279/funnelcake40/win32/en-US/Firefox%20Setup%2039.0.exe
http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/39.0-candidates/build6/partner-repacks-bug1184279/funnelcake41/win32/en-US/Firefox%20Setup%2039.0.exe

Once I get an signoff on them I'll move them to a production location, and set up download.m.o.

Comment 17

[Chris More [:cmore]]

Thanks for the help, Nick!

Comment 18

[Francesco Polizzi [:fpolizzi]]

Thanks for the speedy responses, Nick and Shane!

:nthomas, what does the timeline look like for sign-off and the move to production?

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

By signoff I mean cmore/fpolizzi/mixedpuppy taking a look at the builds, and confirm they're working as expected. After that only 30 minutes, modulo noticing the signoff and timezones.

Comment 20

[Nick Thomas [:nthomas] (UTC+12)]

Bouncer (download.m.o) is configured, but won't work until signoff and the builds are copied. The urls will be:

https://download.mozilla.org/?product=firefox-39.0-SSL-f40&os=win&lang=en-US
https://download.mozilla.org/?product=firefox-39.0-SSL-f41&os=win&lang=en-US

Comment 21

[Nick Thomas [:nthomas] (UTC+12)]

RelEng, since I'm likely be out when signoff comes, we need to do this

# ffxbld@stage
rsync -av /pub/mozilla.org/firefox/candidates/39.0-candidates/build6/partner-repacks-bug1184279/funnelcake40/ /pub/mozilla.org/firefox/releases/39.0-funnelcake40/

rsync -av /pub/mozilla.org/firefox/candidates/39.0-candidates/build6/partner-repacks-bug1184279/funnelcake41/ /pub/mozilla.org/firefox/releases/39.0-funnelcake41/

Wait 5 minutes for sentry find the files, then make sure the two links above work.

Comment 22

[Chris More [:cmore]]

Francesco and I will test tomorrow to see if the variations have the desired settings. 

:nthomas: do these configs use the new versions that mconnor fixed with the default search settings in the US?

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

Yes, his fix was in 38.0.6, 39.0, and will be in releases going forward.

Comment 24

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #23)
> Yes, his fix was in 38.0.6, 39.0, and will be in releases going forward.

Ok, great. thanks

Comment 25

[Francesco Polizzi [:fpolizzi]]

I've tested these builds and they are both working as expected.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Builds published, and urls in comment #20 verified working.

Comment 27

[Francesco Polizzi [:fpolizzi]]

Funnelcakes are in distribution as of 7/20.

Thank you very much :nthomas, :mixedpuppy!

Comment 28

[Francesco Polizzi [:fpolizzi]]

Funnelcakes distributed starting 7/20 and ending 7/28.

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

Cleaned up the config with https://hg.mozilla.org/build/partner-repacks/rev/8100c17d0534 (removed 39 and 40, stashed 41 in inactive-configs).