1202786 - Create funnelcake builds for OSX EN with a 2nd Tab at FirstRun.

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[Alex Davis [:adavis]]

Build:
* language: en
* OSX
* version: Latest Release (currently: 40.0.3)

hhabstritt and adavis are running a UX test where 2 tabs open at firstrun. We plan to measure the impact on FXA creation of having 2 tabs at firstrun and the impact on retention of different value propositions in the 2nd tab. 

This test also serves the purpose of confirming that 2 tabs is a viable channel to communicate value propositions. It needs to be done before the Fall Campaign which is scheduled to go live with FX42. 2 tabs are planned for that campaign but there currently isn't much information available on potential impact (positive or negative).

We need 3 funnelcakes:
1) Control (no change, just regular firstrun)
2) Firstrun opens a second tab with this address: https://www.mozilla.org/en-US/firefox/desktop/customize/?f=4X
3) Same as 2) but the parameter at the end of the URL should be different due to the funnelcake build number.

According to NThomas, we can open a second tab at first run similarly to how we do it for a homepage.

The setting for firstrun pref should look like this:
<url1>|<url2>     
e.g.: 
https://www.mozilla.org/en-US/firefox/40.0/firstrun/|https://www.mozilla.org/en-US/firefox/desktop/customize/

We are unsure if the ?f=4X will automatically be appended to the URL of the 2nd tab in the same manner as it is for the first tab. If it isn't, please just add it manually to the URL so we can detect both funnelcake builds. Depending on the URL parameter we detect, we will dynamically display different content on the 2nd tab.

Additional details about this test are available here (hypothesis, screenshots, future results): 
https://mana.mozilla.org/wiki/display/FIREFOX/Two+Tabs+at+FirstRun

Thanks in advance for your help.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

* I'm assuming you mean en-US for the locale
* funnelcakes 47, 48, 49 have been allocated on https://docs.google.com/spreadsheets/d/1gZ-GFyubAAlIxh-u-5SR52rqKKfBRLRuwrjSq97QkWA/edit#gid=0. Please fill in the distribution details
* config incoming ...

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Pull request for configs

* 47 is the control; 48 & 49 have two first run tabs (with the &f=4X parameters hard wired because there's no code to add them at runtime)
* I've left out startup.homepage_override_url (the whatsnew page), since it's not set in vanilla builds and will be overridden by what the update server says anyway

Comment 3

[Alex Davis [:adavis]]

Yes, correct. en-us!

I added the distribution details to the spreadsheet. Thanks for populating it too. :)

Just to confirm (call it paranoia), please make sure we don't leave them as f=4X but rather their respective funnelcakes numbers (f=48 and f=49).

Thanks so much!

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

I'll spin these up on Monday, if no-one beats me to it.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Builds created, you can find them in 
  http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/

Please verify they're working as expected.

Comment 6

[Alex Davis [:adavis]]

@Rebecca, Holly and I are hoping you could quickly test this for us. We have 3 funnelcake builds:

*Funnelcake 47, control. No change

*Funnelcake 48, opens 2 tabs at firstrun. The second tab should open https://www.mozilla.org/en-US/firefox/desktop/customize/?f=48

*Funnelcake 49, opens 2 tabs at firstrun. The second tab should open https://www.mozilla.org/en-US/firefox/desktop/customize/?f=49

These are osx en language full installers.

We can do 6 verifies, 2 for each funnelcake, with and without profiles.

==First verify==

1) Ensure there is no Firefox profile on the osx machine.

2) Download this: http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/funnelcake48/mac/en-US/

3) Install Firefox

4) Run Firefox

Verify: Verify that https://www.mozilla.org/en-US/firefox/desktop/customize/?f=48 opens in a second tab. Users should see the Firefox account tab by default though.

Content of 2nd tab:
https://mana.mozilla.org/wiki/download/attachments/50266410/Image%202015.08.28%201_02_33%20PM.png?version=1&modificationDate=1440792675892&api=v2

==Second verify==

1) Ensure there is an existing Firefox install on the machine (but not version from above)

2) Download this: http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/funnelcake48/mac/en-US/

3) Install Firefox

4) Run Firefox

Verify: Verify that https://www.mozilla.org/en-US/firefox/desktop/customize/?f=48 DOES NOT open in a second tab. Normally users will see the what's new page.

==Third verify==

1) Ensure there is no Firefox profile on the osx machine.

2) Download this: http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/funnelcake49/mac/en-US/

3) Install Firefox

4) Run Firefox

Verify: Verify that https://www.mozilla.org/en-US/firefox/desktop/customize/?f=49 opens in a second tab. Users should see the Firefox account tab by default though.

Content of 2nd tab:
https://mana.mozilla.org/wiki/download/attachments/50266410/Image%202015.08.28%201_01_42%20PM.png?version=1&modificationDate=1440792643130&api=v2

==Forth verify==

1) Ensure is an existing Firefox install on the machine (but not version from above)

2) Download this: http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/funnelcake49/mac/en-US/

3) Install Firefox

4) Run Firefox

Verify: Verify that https://www.mozilla.org/en-US/firefox/desktop/customize/?f=49 DOES NOT open in a second tab. Normally users will see the what's new page.

==Fifth verify==
1) Ensure there is no Firefox profile on the osx machine.

2) Download this: http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/funnelcake47/mac/en-US/

3) Install Firefox

4) Run Firefox

Verify: Just the regular firstrun with only 1 tab

==Sixth verify==

1) Ensure is an existing Firefox install on the machine (but not version from above)

2) Download this: http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/funnelcake47/mac/en-US/

3) Install Firefox

4) Run Firefox

Verify: Just the regular firstrun with only 1 tab

Comment 7

[Alex Davis [:adavis]]

@Anthony, Rebecca is unfortunately busy all day tomorrow with team meetings in Mountain View, any chance you could help us out with this one with the above mentioned tests?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

The 2nd, 4th, and 6th tests need an existing Firefox profile, with the default pref of not opening windows and tabs from last time to make life easier. In the 6th test it would be the default start page rather than first run, no ? FWIW, I didn't configure any whatsnew pages.

Comment 9

[Alex Davis [:adavis]]

(In reply to Nick Thomas [:nthomas] from comment #8)
> The 2nd, 4th, and 6th tests need an existing Firefox profile, with the
> default pref of not opening windows and tabs from last time to make life
> easier. In the 6th test it would be the default start page rather than first
> run, no ? FWIW, I didn't configure any whatsnew pages.

Thanks for clarifying Nick. That's fine by me. :)

Comment 10

[u279076]

All tests ran as expected except for the fifth test.

> 1) Ensure there is no Firefox profile on the osx machine.
> 2) Download http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/funnelcake47/mac/en-US/
> 3) Install Firefox
> 4) Run Firefox
>
> Verify: Just the regular firstrun with only 1 tab

I get two tabs on first run:
1) https://www.mozilla.org/en-US/firefox/40.0.3/firstrun/?f=47
2) about:home

Comment 11

[Alex Davis [:adavis]]

(In reply to Anthony Hughes, QA Mentor (:ashughes) from comment #10)
> All tests ran as expected except for the fifth test.
> 
> > 1) Ensure there is no Firefox profile on the osx machine.
> > 2) Download http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/40.0.3-candidates/build1/partner-repacks-bug1202786/funnelcake47/mac/en-US/
> > 3) Install Firefox
> > 4) Run Firefox
> >
> > Verify: Just the regular firstrun with only 1 tab
> 
> I get two tabs on first run:
> 1) https://www.mozilla.org/en-US/firefox/40.0.3/firstrun/?f=47
> 2) about:home

Thanks Anthony!

The mistake may be on my side. Is that the default behavior for release? (/firstrun/ & about:home in 2 different tabs) If so, then I think we're ok.

Essentially, tests 5 & 6 are on the control.

Thanks again,

Comment 12

[u279076]

(In reply to Alex Davis [:adavis] from comment #11)
> The mistake may be on my side. Is that the default behavior for release?
> (/firstrun/ & about:home in 2 different tabs) If so, then I think we're ok.

Yes it is, so I think we are okay.

Comment 13

[Alex Davis [:adavis]]

Thanks everyone for the help. We're going to ship this out!

Comment 14

[Alex Davis [:adavis]]

I jumped the gun. ;)

@Nick, let me know when it's on the servers.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

The builds are published, bouncer links are 

https://download.mozilla.org/?product=firefox-40.0.3-SSL-f47&os=osx&lang=en-US
https://download.mozilla.org/?product=firefox-40.0.3-SSL-f48&os=osx&lang=en-US
https://download.mozilla.org/?product=firefox-40.0.3-SSL-f49&os=osx&lang=en-US

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

How long will this study run for ? Will we need to generate 41.0-based funnelcakes ?

Comment 17

[Alex Davis [:adavis]]

Thanks for the links!

I usually let it run for 2 weeks but 41 is 1 week away. I'll see if I can expose these variations to a higher concentration of people to get it out of the way within 1 week.

Let me get back to a little later today or tomorrow.

Comment 18

[Alex Davis [:adavis]]

It's live.

:nthomas, we should normally be able to distribute it to enough people before the next release. We're really aggressive with the distribution of the funnelcakes to get enough of a sample size.

Short answer: we don't think you'll have to make any new funnelcakes for 41.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

I moved the configs into old/ with https://github.com/mozilla-partners/funnelcake/commit/b909125a0d26b353e181c214bdc04428e2c7e8c1, so that we don't generate any more of these by accident.