1143116 - Don't use the modal default browser prompt on first run

Status: RESOLVED DUPLICATE of bug 1367073

(Firefox :: General, defect, P2)

Description

[Verdi [:verdi] Best to slack me]

As outlined in Bug 1126319 we want to move the default browser prompt, shown on first run, into the migrator. This will keep it from interfering with the tour and ensure that everyone who needs to see it does. In this case we won't need to use the modal default prompt on first run (though it can still be shown on a subsequent start up).

Comment 1

[Verdi [:verdi] Best to slack me]

Another option would be to not add this to the migrator and therefor not show it at all on first run. We're waiting on collecting data from the funnelcake in Bug 1191060 before deciding how to move forward.

Comment 2

[:Gijs (he/him)]

(In reply to Verdi [:verdi] from comment #1)
> Another option would be to not add this to the migrator and therefor not
> show it at all on first run. We're waiting on collecting data from the
> funnelcake in Bug 1191060 before deciding how to move forward.

Was there a decision based on this data?

Comment 3

[Verdi [:verdi] Best to slack me]

(In reply to :Gijs Kruitbosch from comment #2)
> (In reply to Verdi [:verdi] from comment #1)
> > Another option would be to not add this to the migrator and therefor not
> > show it at all on first run. We're waiting on collecting data from the
> > funnelcake in Bug 1191060 before deciding how to move forward.
> 
> Was there a decision based on this data?

Not yet. We had to re-run the test and it's been collecting data for about six weeks now. So we should know more soon.

Comment 4

[Jared Wein [:jaws] (please needinfo? me)]

Hey Chris, have you gotten enough results from the funnelcake builds to know how much the default browser dialog affects usage of Firefox (daily active users, monthly active users, usage time of Firefox per day, percentage of users who make Firefox their default)?

Comment 5

[Chris More [:cmore]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #4)
> Hey Chris, have you gotten enough results from the funnelcake builds to know
> how much the default browser dialog affects usage of Firefox (daily active
> users, monthly active users, usage time of Firefox per day, percentage of
> users who make Firefox their default)?

I only have the first 3 weeks of the cohort life and I may get some more data and I may get some more data. What I can say thus far is that there is no change (positive or negative) to retention with the current default browser prompt and no default browser prompt on first run.

That's interesting.

if it is true, this means that the initial prompt to set Firefox as default is not a lever for retention. It doesn't mean it is absolute zero impact, but what it could mean is that there are larger drivers to retention and the impact of the prompt on first run is not enough to alter the overall retention trend.

Comment 6

[Jared Wein [:jaws] (please needinfo? me)]

Hey Chris, can you give an update with any new info you may have?

Comment 7

[Chris More [:cmore]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #6)
> Hey Chris, can you give an update with any new info you may have?

Nothing yet. I have new telemetry data for retention that is still being tweaked the past few days, but it is missing all of the funnelcake data. This was discussed and resolved in bug 1246722. It looks like the funnelcake data is now in another variable. I may have something in the next two weeks if all the work goes well with Mark Reid's data work.

Comment 8

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Chris More [:cmore] from comment #7)
> (In reply to Jared Wein [:jaws] (please needinfo? me) from comment #6)
> > Hey Chris, can you give an update with any new info you may have?
> 
> Nothing yet. I have new telemetry data for retention that is still being
> tweaked the past few days, but it is missing all of the funnelcake data.
> This was discussed and resolved in bug 1246722. It looks like the funnelcake
> data is now in another variable. I may have something in the next two weeks
> if all the work goes well with Mark Reid's data work.

Yikes, I hope it wasn't too much work to rewrite the queries due to bug 1246722. :cmore/:mreid, can you give me some steps for how I can help with running these queries?

Comment 9

[Chris More [:cmore]]

Hi Jared.

I looked into Unified Telemetry data for the funnelcake 60 and 61 cohorts and the data isn't complete and I am unable to calculate retention as I am missing week zero on all the acquisition weeks. It was launched right around when FHR was moving to UT. 

If we could query UT for calculating total usage hours over time, that should potentially show something since it doesn't require calculating percentages based off of week zero profiles.

We are checking with :mreid to see how difficult it would be to create an additional cohort datasource that uses total hours over time to complement the retention dashboard.

Comment 10

[Mark Reid [:mreid]]

We're adding usage hours over in bug 1253751, so hopefully we'll have the required data soon.

Comment 11

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Mark Reid [:mreid] from comment #10)
> We're adding usage hours over in bug 1253751, so hopefully we'll have the
> required data soon.

Hey Mark, do we have enough data now to proceed here?

Comment 12

[Mark Reid [:mreid]]

Alessio, do you have any more details on the progress with the data set?

Maybe we should just consider doing a one-off analysis to get this bug resolved.

Comment 13

[Alessio Placitelli [:Dexter]]

(In reply to Mark Reid [:mreid] from comment #12)
> Alessio, do you have any more details on the progress with the data set?
> 
> Maybe we should just consider doing a one-off analysis to get this bug
> resolved.

In bug 1253751 comment 27 a report was produced using the cohort data, I'm afraid I don't know more than that. I'm not sure that means the data can be used. :cmore probably knows more about that.

If we have a specific set of requirements, we can attack this bug with an ad-hoc analysis. Unfortunately, I won't be able to work on that until mid May.

Comment 14

[Jared Wein [:jaws] (please needinfo? me)]

Mark, can you do a one-off analysis? The questions that were asked in bug 1253751 don't apply here. We are looking to answer two primary questions:

1) Does the usage of the "default browser" prompt shown during startup affect default browser rates.
2) Do we know how much the default browser dialog affects usage of Firefox (daily active users, monthly active users, usage time of Firefox per day, percentage of users who make Firefox their default)?

Comment 15

[Thomas Huelbert]

Saptarshi - are you able to answer the questions posed here?

Comment 16

["Saptarshi Guha[:joy]"]

I think an erstwhile employee Christina Choi answered something related to this (i think on Nightly),  not sure. Currently my plate is quite full.

Comment 17

[Verdi [:verdi] Best to slack me]

(In reply to "Saptarshi Guha[:joy]" from comment #16)
> I think an erstwhile employee Christina Choi answered something related to
> this (i think on Nightly),  not sure. Currently my plate is quite full.

Yes (Bug 1129070), though for some reason we seem unwilling to trust it and are thus trying to answer the question again - Bug 1143116 comment10

Comment 18

[Chris More [:cmore]]

Update!

As previously discussed, the funnelcake test that was ran to test the impact of no default prompt vs the first run default prompt was lost due to the switch from FHR to Unified Telemetry. Well, there is one piece of data that has been consistent from the start of time. That is the ADI.

I looked at the ADIs of the two funnelcakes and the funnelcake with the default browser prompt on first run has a consistently 5% higher ADI during the week. This 5% improvement started on day 1 and continues up until now. While this isn't retention data, ADIs are more tightly linked to usage given the non-unique pings. 

This can be seen here: http://cl.ly/1M2H3z26011V

Funnelcake 60: no prompt - blue
Funnelcake 61: first run prompt (control) - orange

This test doesn't conclude that first run is better than anything else. It just appears from this ADI data that first run is better than doing nothing.

Comment 19

[Mark Reid [:mreid]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #14)
> Mark, can you do a one-off analysis?
Sorry, I don't have bandwidth or this analysis.

Comment 20

[Mark Reid [:mreid]]

*for

Comment 21

[Brian Grinstead [:bgrins]]

It appears this is the current behavior in Nightly via browser.shell.skipDefaultBrowserCheckOnFirstRun (blame points to Bug 1322723). Is that riding the train, and if so can this bug be resolved?

Comment 22

[:Gijs (he/him)]

(In reply to Brian Grinstead [:bgrins] from comment #21)
> It appears this is the current behavior in Nightly via
> browser.shell.skipDefaultBrowserCheckOnFirstRun (blame points to Bug
> 1322723). Is that riding the train, and if so can this bug be resolved?

Yes, this landed in bug 1367073 and is on for 55.