1202083 - Understand how the /firstrun/ page impacts product usage and retention

Status: NEW

(Firefox :: General, defect)

Description

[Chris More [:cmore]]

In bug 1149684 (FxA button) and bug 1150222 (FxA form), we implemented new First Run pages that suggests to new users that creating a Firefox Account, sync'ing Firefox's data, and getting Firefox on mobile, is something positive for the user experience. 

With that, we would like to understand better how the new first run page impacts product usage and retention in the early days/weeks of a new user experiencing the product. 

We could do a funnelcake tests with two cohorts, one with the current /firstrun/ page, and a second without a /firstrun/ page. Everything else between the two chorts will be the same.

Funnelcake A:

pref("startup.homepage_welcome_url","https://www.mozilla.org/%LOCALE%/firefox/%VERSION%/firstrun/");

Funnelcake B:

pref("startup.homepage_welcome_url","");

If we wanted to compare the a product /firstrun/ to a non-product message, we could do a third funnelcake like:

Funnelcake C:

pref("startup.homepage_welcome_url","https://www.mozilla.org/%LOCALE%/mission/");

Funnelcake info:

* Version: TBD
* OS: win32
* Lang: en-US

Comment 1

[Francesco Polizzi [:fpolizzi]]

Submitted a pull request on Github for the configs for these three funnelcake configs (50,51,52) and added their meta data to the funnelcake master tracker.

https://github.com/mozilla-partners/funnelcake/pull/2

Comment 2

[Shane Caraveo (:mixedpuppy)]

Francesco, see my comments in the pull request.

@cmore, also, should funnelcake C have ?f=52

Comment 3

[Shane Caraveo (:mixedpuppy)]

changes are merged, ready for builds

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Chris More [:cmore] from comment #0)
> Funnelcake info:
> 
> * Version: TBD
> * OS: win32
> * Lang: en-US

Lets resolve the TBD here.

Comment 5

[Chris More [:cmore]]

Sorry for the delay.

Version: 41.0

(In reply to Shane Caraveo (:mixedpuppy) from comment #2)
> Francesco, see my comments in the pull request.
> 
> @cmore, also, should funnelcake C have ?f=52

Yes

A:

pref("startup.homepage_welcome_url","https://www.mozilla.org/%LOCALE%/firefox/%VERSION%/firstrun/?f=50");

C:

pref("startup.homepage_welcome_url","https://www.mozilla.org/%LOCALE%/mission/?f=52");

Comment 6

[Francesco Polizzi [:fpolizzi]]

If version doesn't affect the config files, we should be good to go ahead with the builds.

Look good @mixedpuppy?

Comment 7

[Shane Caraveo (:mixedpuppy)]

Yeah, I think it looks good.  @nthomas or @coop for builds?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to francescosapolizzi from comment #6)
> If version doesn't affect the config files, we should be good to go ahead
> with the builds.

Which should be based on 41.0, given that's going to ship next week ? That's the kind of clarity we need here.

Comment 9

[Francesco Polizzi [:fpolizzi]]

(In reply to Nick Thomas [:nthomas] from comment #8)
> (In reply to francescosapolizzi from comment #6)
> > If version doesn't affect the config files, we should be good to go ahead
> > with the builds.
> 
> Which should be based on 41.0, given that's going to ship next week ? That's
> the kind of clarity we need here.

Yes, I believe we want to base the build on 41.0.

Comment 10

[Chris Cooper [:coop] (he/him)]

Shane/Nick: did the builds get created here, or are you still waiting on them? I can create them today if necessary.

Comment 11

[Shane Caraveo (:mixedpuppy)]

afaik the builds are not done, they are funnelcakes 50,51,52 in the github repo.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

I haven't created them, if you're willing then go for it.

Comment 13

[Francesco Polizzi [:fpolizzi]]

Any update on this :coop?

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Francesco Polizzi [:fpolizzi] from comment #13)
> Any update on this :coop?

Builds are here for verification:

http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks-bug1202083/funnelcake50/
http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks-bug1202083/funnelcake51/
http://stage.mozilla.org/pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks-bug1202083/funnelcake52/

I've also added them to bouncer based on Nick's instructions in https://bugzilla.mozilla.org/show_bug.cgi?id=1195843#c21

Remaining step after verification is to run the rsync:

# ffxbld@stage
 rsync -av /pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks-bug1202083/funnelcake50/ /pub/mozilla.org/firefox/releases/41.0-funnelcake50/
rsync -av /pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks-bug1202083/funnelcake51/ /pub/mozilla.org/firefox/releases/41.0-funnelcake51/
 rsync -av /pub/mozilla.org/firefox/candidates/41.0-candidates/build3/partner-repacks-bug1202083/funnelcake52/ /pub/mozilla.org/firefox/releases/41.0-funnelcake52/

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

We'll need to regenerate these now that we shipped Firefox 41.0.1 ?

Comment 16

[Chris Cooper [:coop] (he/him)]

(In reply to Nick Thomas [:nthomas] from comment #15)
> We'll need to regenerate these now that we shipped Firefox 41.0.1 ?

I've put new funnelcakes for 41.0.1 here:

http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/partner-repacks-bug1202083/funnelcake50/
http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/partner-repacks-bug1202083/funnelcake51/
http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/partner-repacks-bug1202083/funnelcake52/

Nick: can I re-use the existing products in bouncer and just modify the paths, or should I create fresh product entries for 41.0.1?

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

We'll need to re-add with products like Firefox-41.0.1-SSL-f50. IIRC on mozilla.org the -f50 is appended to the latest version in product details.

Comment 18

[Francesco Polizzi [:fpolizzi]]

(In reply to Nick Thomas [:nthomas] from comment #17)
> We'll need to re-add with products like Firefox-41.0.1-SSL-f50. IIRC on
> mozilla.org the -f50 is appended to the latest version in product details.


Should these builds okay for QA in their current form?

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Yes, the builds in comment #16 are ready for testing. Bouncer is just about high-traffic distribution when it's time to ship.

Comment 20

[Matt Brandt [:mbrandt]]

QA verified - Win7 Pro x32 -
- A fresh profile for each build displays the expected startup page
- startup.homepage_welcome_url mirrors what is expected for each build

Funnelcake A:
pref("startup.homepage_welcome_url","https://www.mozilla.org/%LOCALE%/firefox/%VERSION%/firstrun/");

http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/partner-repacks-bug1202083/funnelcake50/

Funnelcake B:

pref("startup.homepage_welcome_url","");

http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/partner-repacks-bug1202083/funnelcake51/

Funnelcake C:

pref("startup.homepage_welcome_url","https://www.mozilla.org/%LOCALE%/mission/");

http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-candidates/build2/partner-repacks-bug1202083/funnelcake52/

Comment 21

[Francesco Polizzi [:fpolizzi]]

(In reply to Matt Brandt [:mbrandt] from comment #20)
> QA verified - Win7 Pro x32 -
> - A fresh profile for each build displays the expected startup page
> - startup.homepage_welcome_url mirrors what is expected for each build
> 
> Funnelcake A:
> pref("startup.homepage_welcome_url","https://www.mozilla.org/%LOCALE%/
> firefox/%VERSION%/firstrun/");
> 
> http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-
> candidates/build2/partner-repacks-bug1202083/funnelcake50/
> 
> Funnelcake B:
> 
> pref("startup.homepage_welcome_url","");
> 
> http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-
> candidates/build2/partner-repacks-bug1202083/funnelcake51/
> 
> Funnelcake C:
> 
> pref("startup.homepage_welcome_url","https://www.mozilla.org/%LOCALE%/
> mission/");
> 
> http://archive.mozilla.org/pub/mozilla.org/firefox/candidates/41.0.1-
> candidates/build2/partner-repacks-bug1202083/funnelcake52/

Thanks, Matt! I have also confirmed that these builds are working on x32 Windows 8.

:Coop, looks like we are ready to go with these builds. Can we get these ready for distribution?

Comment 22

[Chris Cooper [:coop] (he/him)]

(In reply to Francesco Polizzi [:fpolizzi] from comment #21) 
> :Coop, looks like we are ready to go with these builds. Can we get these
> ready for distribution?

Added new bouncers entries and rsync-ed the repacks to the release directory.

Comment 23

[Francesco Polizzi [:fpolizzi]]

Distribution started on 10/14/15

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

FYI 41.0.2 is coming, probably shipping tomorrow. Will distribution be complete by then ? If not, please reopen this bug for us to regenerate the funnelcakes.

Comment 25

[Francesco Polizzi [:fpolizzi]]

(In reply to Nick Thomas [:nthomas] from comment #24)
> FYI 41.0.2 is coming, probably shipping tomorrow. Will distribution be
> complete by then ? If not, please reopen this bug for us to regenerate the
> funnelcakes.

Do you think the update warrants stopping the release of this funnelcake in 41.0.1? If so, we will stop the distribution and rebuild with 41.0.2

Comment 26

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #24)
> FYI 41.0.2 is coming, probably shipping tomorrow. Will distribution be
> complete by then ? If not, please reopen this bug for us to regenerate the
> funnelcakes.

Can you re-create these funnelcakes with the same IDs and everything, but just 41.0.2 versions of Firefox and put up on download.mozilla.org? If you can keep it all the same and get it live, I can increment the version number in the experiment. We need to keep this running for another two weeks. Thanks!

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Builds regenerated, products & locations added in bouncer, pushed, and 
  https://download.mozilla.org/?product=firefox-41.0.2-SSL-f50&os=win&lang=en-US
  https://download.mozilla.org/?product=firefox-41.0.2-SSL-f51&os=win&lang=en-US
  https://download.mozilla.org/?product=firefox-41.0.2-SSL-f52&os=win&lang=en-US
are working.

As ever, I encourage someone to verify it's working as expected.

Comment 28

[Chris More [:cmore]]

(In reply to Nick Thomas [:nthomas] from comment #27)
> Builds regenerated, products & locations added in bouncer, pushed, and 
>  
> https://download.mozilla.org/?product=firefox-41.0.2-SSL-f50&os=win&lang=en-
> US
>  
> https://download.mozilla.org/?product=firefox-41.0.2-SSL-f51&os=win&lang=en-
> US
>  
> https://download.mozilla.org/?product=firefox-41.0.2-SSL-f52&os=win&lang=en-
> US
> are working.
> 
> As ever, I encourage someone to verify it's working as expected.

Matt: can you re-perform your QA you did in comment 20, but with these links? The sooner the better as Firefox had a chemspill and we have to re-ship these funnelcakes.

/firstrun/?f=50

https://download.mozilla.org/?product=firefox-41.0.2-SSL-f50&os=win&lang=en-US

No first run

https://download.mozilla.org/?product=firefox-41.0.2-SSL-f51&os=win&lang=en-US

/mission/?f=52

https://download.mozilla.org/?product=firefox-41.0.2-SSL-f52&os=win&lang=en-US

Thanks!

Comment 29

[Matt Brandt [:mbrandt]]

QA verified - Win8.1 x32 -
- A fresh profile for each build displays the expected startup page
- startup.homepage_welcome_url mirrors what is expected for each build

Funnelcake A:
pref("startup.homepage_welcome_url","https://www.mozilla.org/en-US/%APP%/%VERSION%/firstrun/?f=50");

https://download.mozilla.org/?product=firefox-41.0.2-SSL-f50&os=win&lang=en-US

Funnelcake B:

pref("startup.homepage_welcome_url","");

https://download.mozilla.org/?product=firefox-41.0.2-SSL-f51&os=win&lang=en-US

Funnelcake C:

pref("startup.homepage_welcome_url","https://www.mozilla.org/en-US/mission?f=52");

https://download.mozilla.org/?product=firefox-41.0.2-SSL-f52&os=win&lang=en-US

Comment 30

[Francesco Polizzi [:fpolizzi]]

(In reply to Matt Brandt [:mbrandt] from comment #29)
> QA verified - Win8.1 x32 -
> - A fresh profile for each build displays the expected startup page
> - startup.homepage_welcome_url mirrors what is expected for each build
> 
> Funnelcake A:
> pref("startup.homepage_welcome_url","https://www.mozilla.org/en-US/%APP%/
> %VERSION%/firstrun/?f=50");
> 
> https://download.mozilla.org/?product=firefox-41.0.2-SSL-f50&os=win&lang=en-
> US
> 
> Funnelcake B:
> 
> pref("startup.homepage_welcome_url","");
> 
> https://download.mozilla.org/?product=firefox-41.0.2-SSL-f51&os=win&lang=en-
> US
> 
> Funnelcake C:
> 
> pref("startup.homepage_welcome_url","https://www.mozilla.org/en-US/
> mission?f=52");
> 
> https://download.mozilla.org/?product=firefox-41.0.2-SSL-f52&os=win&lang=en-
> US

QA verified from my end also. Ready to start optimizely test once the links are updated on the test. I didn't want to pause the current test and we need to do that in order to change links. Cmore, over to you!

Comment 31

[Chris More [:cmore]]

Links all updated and I didn't have to stop anything. Thanks everyone.

Comment 32

[Matt Brandt [:mbrandt]]

(In reply to Chris More [:cmore] from comment #31)
> Links all updated and I didn't have to stop anything. Thanks everyone.

\0/

Comment 33

[Chris More [:cmore]]

Funnelcake shut off. We got enough data. 40k downloads per variation. That should be enough to see if there is any impact to retention among the 3 cohorts. We won't have the answer to this for probably another 5-10 weeks.