1191242 - Instrument the default browser prompt

Status: VERIFIED FIXED

(Firefox :: Shell Integration, defect, P1)

Description

[(Currently slow to respond) Philipp Sackl [:phlsa] (Firefox UX) please use needinfo]

As we are working on improving our first run experience, it would be tremendously helpful to get a better understanding of how users interact with the default browser prompt.

Here's what we'd like to know:
- How often does the dialog show up per profile?
- What's the distribution of clicks on the buttons in the dialog?
- How often does the "don't ask me again" box get checked?
- How often is setting the default browser successful (on Win8/10 where users have to take extra steps after clicking the button)

Bryan, Dolske, did I miss anything?

Comment 1

[Bryan Clark (DevTools PM) [:clarkbw]]

Looks like the right list.

Comment 2

[Justin Dolske [:Dolske]]

Refactoring and expanding the list, as follows...

Basic goal: We want to understand how the default browser prompt's impact and if it's not working for people.

- Does the user have "Always check if Firefox is your default browser" pref enabled (startup check for browser.shell.checkDefaultBrowser)

- User action performed when the prompt is shown. There are three possibilities:
  1: Not now + "Always perform this check" enabled
  2: Not now + "Always perform this check" disabled
  3: "Use Firefox as my default" clicked

  * We don't distinguish between clicking Not Now / pressing ESC / clicking X
  * Curiously, we don't update "Always perform" checkbox pref (if it was changed) when setting Firefox
    as the default. If we fix that, case #3 splits into two ala #1 / #2.
  * This doesn't directly record if the user _changed_ the checkbox, but we can indirectly infer that by
    comparing with the previous probe.
  * Since it's a modal prompt, the sum of these tells us how often the prompt is shown. (Well, unless
    the user never interacts with Firefox again, but I'd expect that to be very rare.)

- Did SetDefaultBrowser() encounter any errors?
  * There's a surprising amount of code that runs here, and numerous possible failure points.
  * I'd settle for just a root-level pass/fail, and we can dig in more if the failure rate is
    significantly above 0. But a probe enumerating the first level or so of possible failure types
    might be just as easy?

- Did IsDefaultBrowser() encounter any errors?
  * As above; this code is surprisingly complex and has numerous possible failure points.

- How long did it take the user to select Firefox as their default in the Windows UI?
  * This is the complex one.  On Win 8/10 we don't currently get any kind of callback or way of seeing
    how the user is faring in the native app-chooser. Maybe worth investigating, but it's probably
    simplest to just poll on an interval to see if we're the default app. Say, check every 1 second until
    either we're the default (success!) or 120 seconds go by (user is hopelessly lost).
  * TODO: consider how/if to handle crashes, quitting the app, or the user trying again.
  * This probe could be Win8/10 specific, since it should be immediate everywhere else. OTOH, if we do it
    everywhere, we can check to see if it ever silently fails on other platforms.

We already have a BROWSER_IS_USER_DEFAULT probe that reports if Firefox is the user's default browser at startup. (As well as having it measured in FHR.)

Given the importance of this data (detecting errors, user confusion, and the baselines to understand their relative frequency) these should all be opt-out telemetry probes.

Comment 3

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch

To test this patch, make sure that you have `toolkit.telemetry.enabled` set to true in about:config.

Comment 4

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch v1

(updated patch to use make these opt-out on release channel)

Comment 5

[Jared Wein [:jaws] (please needinfo? me)]

The attached patch adds new probes that are opt-out on release channel.

These are the following probes:
 "BROWSER_IS_USER_DEFAULT_ERROR": {
   "description": "True if the browser was unable to determine if the browser was set as default."
 },
 "BROWSER_SET_DEFAULT_DIALOG_PROMPT_RAWCOUNT": {
   "description": "The number of times that a profile has seen the 'Set Default Browser' dialog."
 },
 "BROWSER_SET_DEFAULT_ALWAYS_CHECK": {
   "description": "True if the profile has `browser.shell.checkDefaultBrowser` set to true."
 },
 "BROWSER_SET_DEFAULT_RESULT": {
   "description": "Result of the Set Default Browser dialog (0=Use Firefox + 'Always perform check' unchecked, 1=Use Firefox + 'Always perform check' checked, 2=Not Now + 'Always perform check' unchecked, 3=Not Now + 'Always perform check' checked)"
 },
 "BROWSER_SET_DEFAULT_ERROR": {
   "description": "True if the browser was unable to set Firefox as the default browser"
 },
 "BROWSER_SET_DEFAULT_TIME_TO_COMPLETION_SECONDS": {
   "description": "Time to successfully set Firefox as the default browser after clicking 'Set Firefox as Default'. Should be near-instant in some environments, others require user interaction. Measured in seconds."
 },

And makes the following probe opt-out, to match the probes added by this bug:
   "BROWSER_IS_USER_DEFAULT": {
     "description": "The result of the startup default desktop browser check."
   },

Comment 6

[Jared Wein [:jaws] (please needinfo? me)]

I forgot to say why I tagged you for needinfo on the previous comment, but it was to get your approval on adding these new probes. We would like to use opt-out probes for this since it could be a key factor in retaining users.

Comment 7

[Benjamin Smedberg]

isDefaultBrowser is already available in the UT environment and so I think we should remove BROWSER_IS_USER_DEFAULT entirely:

https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/telemetry/environment.html

Also several of these probes are simply recording the value of various preferences, and should be included in the environment instead of as histograms. You can add pref values to the environment here:
http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/TelemetryEnvironment.jsm#83

Without jumping to conclusions, let me ask the following questions:
* What is the user benefit we are trying to provide by collecting this data?
* What questions are these measurements supposed to answer?
* Who is going to develop the reports to monitor this data after it is collected?
* Who is responsible for owning and monitoring these measurements? How frequently will they be monitored?
* I note that you have set the histograms to never expire. Is that really necessary?

Comment 8

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch v1.1

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #7)
> isDefaultBrowser is already available in the UT environment and so I think
> we should remove BROWSER_IS_USER_DEFAULT entirely:
> 
> https://gecko.readthedocs.org/en/latest/toolkit/components/telemetry/
> telemetry/environment.html

Thanks, with this I removed the telemetry probe for BROWSER_IS_USER_DEFAULT as well as the BROWSER_IS_USER_DEFAULT_ERROR since the telemetry environment probe already provides "null" to indicate a failure when reading this. We could protentially keep the BROWSER_IS_USER_DEFAULT_ERROR here as we may want to record the-occurrences-of-an-error-reading-this-value-during-browser-startup.

> Also several of these probes are simply recording the value of various
> preferences, and should be included in the environment instead of as
> histograms. You can add pref values to the environment here:
> http://mxr.mozilla.org/mozilla-central/source/toolkit/components/telemetry/
> TelemetryEnvironment.jsm#83

Thanks, the "should check default" was also present here as well, so I removed it from the patch.
 
> Without jumping to conclusions, let me ask the following questions:
> * What is the user benefit we are trying to provide by collecting this data?

The default browser on the OS is a strong tool for interoperating with other software present on the users computer. If the browser cannot be set as default then there can be many instances where a user clicks on a link to open a webpage and their desired browser is not opened as a result.

> * What questions are these measurements supposed to answer?

The main question behind these probes is to track how effective the default browser prompt is. We have assumptions that the default-ness of a browser heavily affects usage, and we need to be able to respond to changes in operating systems and environments that may prevent the default browser code from working.

> * Who is going to develop the reports to monitor this data after it is
> collected?

This should be monitored by the Firefox PM, but I have not seen a formal notice regarding that. In its absence, I am fine with generating these reports.

> * Who is responsible for owning and monitoring these measurements? How
> frequently will they be monitored?

Ideally we would monitor these reports at each major OS release as well as periodically (every 36 weeks) in the absence of major OS releases.

> * I note that you have set the histograms to never expire. Is that really
> necessary?

As the "default browser" behavior is central to the behavior of the browser externally through the system, this is something that we want to monitor and track with no end-date. Microsoft has made changes to how default applications work in Windows 8 and in Windows 10, and it is fair to expect more changes from Apple and Microsoft in the future.

Comment 10

[Benjamin Smedberg]

> > * Who is going to develop the reports to monitor this data after it is
> > collected?
> 
> This should be monitored by the Firefox PM, but I have not seen a formal
> notice regarding that. In its absence, I am fine with generating these
> reports.

Great, please file a followup bug for this, and I'll hook you up with people who can show you how to do this.

> 
> > * Who is responsible for owning and monitoring these measurements? How
> > frequently will they be monitored?
> 
> Ideally we would monitor these reports at each major OS release as well as
> periodically (every 36 weeks) in the absence of major OS releases.

I'm surprised that we wouldn't look at the report every two weeks or something. In any case, the question is really who is going to be doing this. A product manager? If so, have we identified which one?

I'm ok with this as long as we can identify the person who's responsible for long-term monitoring.

Comment 11

[Justin Dolske [:Dolske]]

Comment on attachment 8649075 [details] [diff] [review]
Patch v1.1

Review of attachment 8649075 [details] [diff] [review]:
-----------------------------------------------------------------

I think there should also be a probe for "Does the user have "Always check if Firefox is your default browser" pref enabled (startup check for browser.shell.checkDefaultBrowser)". We always check & record if Firefox is the default (IIRC), but it would be good to also record this to understand the relative scale of the other data, when it comes to actually prompting the user.

Sorry for the delay here.

::: browser/components/nsBrowserGlue.js
@@ +1194,5 @@
>        catch (ex) { /* never mind; suppose SessionStore is broken */ }
>  
>        // startup check, check all assoc
> +      let isDefault = false;
> +      let isDefaultError = false;

Looks like |isDefaultError| is unused now, but see comment about environment data.

@@ -1198,4 @@
>        try {
> -        // Report default browser status on startup to telemetry
> -        // so we can track whether we are the default.
> -        Services.telemetry.getHistogramById("BROWSER_IS_USER_DEFAULT")

How can we actually see the telemetry environment data? I don't see anything on telemetry.mozilla.org, and without an easy way for people to check this feels like a step backwards. EG, it's pretty trivial for someone to go check BROWSER_IS_USER_DEFAULT right now.

@@ +1210,5 @@
> +      let willPrompt = shouldCheck && !isDefault && !willRecoverSession;
> +
> +      try {
> +        let promptCount = Services.prefs.getIntPref("browser.shell.defaultBrowserAppearanceCount");
> +        promptCount += willPrompt ? 1 : 0;

I think this is clearer as just:

  if (willPrompt)
    promptCount++;

And, uhh, shouldn't you be storing the new pref value somewhere? Otherwise this will always record 0 or 1.

@@ +2733,5 @@
>  #endif
>      try {
>        ShellService.setDefaultBrowser(claimAllTypes, false);
> +
> +      let win = RecentWindow.getMostRecentBrowserWindow();

nsITimer is the usual way to do this from a component...

@@ +2749,5 @@
> +          isDefaultError = true;
> +        }
> +
> +        let now = Math.floor(Date.now() / 1000);
> +        if (isDefault || now - this._completionStartTime > 600) {

Hmm, seems a little weird to be recording the probe after a timeout. But I guess it doesn't really matter, makes it a tad easier to compare the successes relative to the timeout...

@@ +2867,5 @@
>                          (ps.BUTTON_TITLE_IS_STRING * ps.BUTTON_POS_1) +
>                          ps.BUTTON_POS_0_DEFAULT;
>        let rv = ps.confirmEx(win, promptTitle, promptMessage, buttonFlags,
>                              yesButton, notNowButton, null, askLabel, shouldAsk);
> +      Services.console.logStringMessage(rv);

I assume you meant to remove this. ;-)

::: toolkit/components/telemetry/Histograms.json
@@ +7189,5 @@
>      "description": "The number of edges serialized into a heap snapshot."
>    },
> +  "BROWSER_SET_DEFAULT_DIALOG_PROMPT_RAWCOUNT": {
> +    "expires_in_version": "never",
> +    "kind": "linear",

This seems like it should almost be a "count" kind. But the way you want to use this is a bit odd, because it's a count that spans sessions (and so would need to be initialized to the existing value before incrementing the count). The telemetry docs are kind of vague on how this works, but can you initially .add(prefValue) unconditionally, and then .add(1) when the prompt is actually shown? [I wonder how that's handled for collection/backend, though... Do the counts from all your sessions ever end up summed together? I'm not really sure how this all works.]

A histogram is probably ok in this case, though, because I'd rarely expect the prompt to be shown more than once in a session, and so we'll almost always be recording a single value into the histogram.

@@ +7191,5 @@
> +  "BROWSER_SET_DEFAULT_DIALOG_PROMPT_RAWCOUNT": {
> +    "expires_in_version": "never",
> +    "kind": "linear",
> +    "high": "1000",
> +    "n_buckets": "10",

This seems like a surprising choice.

I'd expect most users to only see this a handful of times, with a long tail of people who get a prompt every run. 1000 linear buckets of 10 doesn't tell us a whole lot, since almost everyone would be grouped in the first 0-100 bucket.

I'd suggest an exponential histogram for 0-250. (Not sure how many buckets -- "enough" to give us reasonable resolution in the most common 0-20ish range?)

@@ +7212,5 @@
> +  "BROWSER_SET_DEFAULT_TIME_TO_COMPLETION_SECONDS": {
> +    "expires_in_version": "never",
> +    "kind": "exponential",
> +    "high": "600",
> +    "n_buckets": "20",

I wish there was an easy way to see what the actual bucket ranges end up being for a given size... Hard to know if the bucket size should be bigger or smaller.

Comment 12

[Justin Dolske [:Dolske]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #10)

> > Ideally we would monitor these reports at each major OS release as well as
> > periodically (every 36 weeks) in the absence of major OS releases.
> 
> I'm surprised that we wouldn't look at the report every two weeks or
> something. In any case, the question is really who is going to be doing
> this. A product manager? If so, have we identified which one?

Well, I think it's a bit of both.

I expect to be monitoring this data pretty closely in the near term because it's a current focus, whereas over the longer term it's more of a metric that we'd monitor at less frequent intervals. But we'd still need historical data for long-term trends and exceptional events (like OS releases or other triggers that would place this topic in the limelight again).

Comment 13

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch v2

(In reply to Justin Dolske [:Dolske] from comment #11)
> @@ -1198,4 @@
> >        try {
> > -        // Report default browser status on startup to telemetry
> > -        // so we can track whether we are the default.
> > -        Services.telemetry.getHistogramById("BROWSER_IS_USER_DEFAULT")
> 
> How can we actually see the telemetry environment data? I don't see anything
> on telemetry.mozilla.org, and without an easy way for people to check this
> feels like a step backwards. EG, it's pretty trivial for someone to go check
> BROWSER_IS_USER_DEFAULT right now.

I spoke with some people working on Telemetry and we currently lack good tools for viewing the Telemetry environment data. One would need to spend a matter of days to create the dashboard to view this data. In the interest of time, I have implemented these as opt-out telemetry histograms, which will provide the same level of coverage but with better views into the data.

> @@ +1210,5 @@
> > +      let willPrompt = shouldCheck && !isDefault && !willRecoverSession;
> > +
> > +      try {
> > +        let promptCount = Services.prefs.getIntPref("browser.shell.defaultBrowserAppearanceCount");
> > +        promptCount += willPrompt ? 1 : 0;

> ::: toolkit/components/telemetry/Histograms.json
> @@ +7189,5 @@
> >      "description": "The number of edges serialized into a heap snapshot."
> >    },
> > +  "BROWSER_SET_DEFAULT_DIALOG_PROMPT_RAWCOUNT": {
> > +    "expires_in_version": "never",
> > +    "kind": "linear",
> 
> This seems like it should almost be a "count" kind. But the way you want to
> use this is a bit odd, because it's a count that spans sessions (and so
> would need to be initialized to the existing value before incrementing the
> count). The telemetry docs are kind of vague on how this works, but can you
> initially .add(prefValue) unconditionally, and then .add(1) when the prompt
> is actually shown? [I wonder how that's handled for collection/backend,
> though... Do the counts from all your sessions ever end up summed together?
> I'm not really sure how this all works.]
> 
> A histogram is probably ok in this case, though, because I'd rarely expect
> the prompt to be shown more than once in a session, and so we'll almost
> always be recording a single value into the histogram.
> 

I switched this probe to a "count" metric, which means we no longer need to track the individual occurrences in a pref. Each time that we add to this probe it will increment by 1 and the resulting histogram will be incremented, resuming from its previous count. This is similar to the ONBEFOREUNLOAD_PROMPT_COUNT probe.

> @@ +7191,5 @@
> > +  "BROWSER_SET_DEFAULT_DIALOG_PROMPT_RAWCOUNT": {
> > +    "expires_in_version": "never",
> > +    "kind": "linear",
> > +    "high": "1000",
> > +    "n_buckets": "10",
> 
> This seems like a surprising choice.
> 
> I'd expect most users to only see this a handful of times, with a long tail
> of people who get a prompt every run. 1000 linear buckets of 10 doesn't tell
> us a whole lot, since almost everyone would be grouped in the first 0-100
> bucket.
> 
> I'd suggest an exponential histogram for 0-250. (Not sure how many buckets
> -- "enough" to give us reasonable resolution in the most common 0-20ish
> range?)

I looked through Histograms.json and "count" probes don't appear to have a concept of buckets.
 
> @@ +7212,5 @@
> > +  "BROWSER_SET_DEFAULT_TIME_TO_COMPLETION_SECONDS": {
> > +    "expires_in_version": "never",
> > +    "kind": "exponential",
> > +    "high": "600",
> > +    "n_buckets": "20",
> 
> I wish there was an easy way to see what the actual bucket ranges end up
> being for a given size... Hard to know if the bucket size should be bigger
> or smaller.

I switched this to high of 500 and n_buckets of 15, which matches the same distribution that FX_THUMBNAILS_CAPTURE_TIME_MS has. This should give you a good idea of the bucket sizes.

Comment 14

[Justin Dolske [:Dolske]]

Comment on attachment 8654405 [details] [diff] [review]
Patch v2

Review of attachment 8654405 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/components/nsBrowserGlue.js
@@ +2787,5 @@
> +          isDefaultError = true;
> +        }
> +
> +        let now = Math.floor(Date.now() / 1000);
> +        if (isDefault || now - this._completionStartTime > 600) {

Using a |let runTime = now - this._completionStartTime| might help improve readability a bit.

@@ +2790,5 @@
> +        let now = Math.floor(Date.now() / 1000);
> +        if (isDefault || now - this._completionStartTime > 600) {
> +          if (this._completionTimer) {
> +            this._completionTimer.cancel();
> +          }

I think the if isn't needed -- you'll always have a completionTimer if the callback is running.

Would be good to also null it out here, just to save a few bytes for the rest of the process lifetime.

Comment 15

[Justin Dolske [:Dolske]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #13)


> > > +  "BROWSER_SET_DEFAULT_DIALOG_PROMPT_RAWCOUNT": {
> > > +    "expires_in_version": "never",
> > > +    "kind": "linear",
> > 
> > This seems like it should almost be a "count" kind. But [...]
> 
> I switched this probe to a "count" metric, which means we no longer need to
> track the individual occurrences in a pref. Each time that we add to this
> probe it will increment by 1 and the resulting histogram will be
> incremented, resuming from its previous count. This is similar to the
> ONBEFOREUNLOAD_PROMPT_COUNT probe.

Well, if you're not persisting the count it's basically always just going to be 1 and we can just remove the probe entirely.

I believe the intent Philipp was going for was to understand if users are seeing the prompt endlessly (across sessions), such that it might actually be an annoyance (see also bug 1191250). A similar question would be "how many times to uses see the prompt before they either accept it or turn it off".

Such a probe will still be useful, but we could also do it in a separate bug. I think that's more of a short-term question/study, as opposed to the other probes being added here. Or you can add it here, I don't care much either way. :)

My "but" comment above was mostly just a commentary on how it's not clear which of a histogram or count type probe should be used for this, and so whoever implements this should look into that issue / correct any misunderstanding I have.

Comment 16

[Jared Wein [:jaws] (please needinfo? me)]

url:        https://hg.mozilla.org/integration/fx-team/rev/acabb9982aca0735e5755960d94736d5a4ef081e
changeset:  acabb9982aca0735e5755960d94736d5a4ef081e
user:       Jared Wein <[e-mail]>
date:       Tue Sep 01 15:52:05 2015 -0400
description:
Bug 1191242 - Instrument the default browser prompt. r=dolske p=bsmedberg

Comment 17

[Benjamin Smedberg]

This appears to have landed without having a clear owner for who's monitoring this long-term. Dolske will be monitoring this short-term and that's great, but unless we also have somebody committed to monitoring this permanently, the data collection should be marked to expire after 6 months.

One or the other must happen.

Comment 18

[Bryan Clark (DevTools PM) [:clarkbw]]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #17)
> This appears to have landed without having a clear owner for who's
> monitoring this long-term. Dolske will be monitoring this short-term and
> that's great, but unless we also have somebody committed to monitoring this
> permanently, the data collection should be marked to expire after 6 months.

I believe I'm the person on point for long term monitoring.  We'll be using this as part of the marketing funnel improvement efforts happening through out the rest of the year.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/acabb9982aca

Comment 20

[Jared Wein [:jaws] (please needinfo? me)]

Clearing needinfo as of comment #18.

Comment 21

[Sheila Mooney]

[Tracking Requested - why for this release]: This is checked into Nightly. I assume jaws will request uplift for beta. Adding the tracking flag.

Comment 22

[Jared Wein [:jaws] (please needinfo? me)]

Comment on attachment 8654405 [details] [diff] [review]
Patch v2

Approval Request Comment
[Feature/regressing bug #]: telemetry tracking for default browser changes (more significant with windows10)
[User impact if declined]: delayed data collection
[Describe test coverage new/current, TreeHerder]: manual verification of telemetry results
[Risks and why]: low risk, telemetry only
[String/UUID change made/needed]: none

Comment 23

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch for Beta41

Approval Request Comment
[Feature/regressing bug #]: telemetry tracking for default browser changes (more significant with windows10)
[User impact if declined]: delayed data collection
[Describe test coverage new/current, TreeHerder]: manual verification of telemetry results
[Risks and why]: low risk, telemetry only
[String/UUID change made/needed]: none

Comment 24

[Jared Wein [:jaws] (please needinfo? me)]

Attachment: Patch for Aurora42

See comment #22.

Comment 25

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8657165 [details] [diff] [review]
Patch for Beta41

Unified Telemetry is pref'd off by default in FF41. The risk associated with this uplift is therefore pretty low. Beta41+

Comment 26

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8657174 [details] [diff] [review]
Patch for Aurora42

Seems safe to uplift to Aurora41 too.

Comment 27

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Tracked as this one seems to be adding a lot of telemetry probes. This will also help us stay aware of any unexpected fallouts.

Comment 28

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/dedd83e6d526

Comment 29

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-beta/rev/d689450368e1

Comment 30

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/releases/mozilla-aurora/rev/6eb32a550136

Comment 31

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I accidentally landed the m-c version of the patch to aurora (didn't catch that there was an aurora-specific patch posted until after the fact). Backed out the original and landed the new one in ^

Comment 32

[Andrei Vaida [:avaida]]

I've indeed noticed the following probes on Firefox 41.0b9 (20150910171927):
 • BROWSER_IS_USER_DEFAULT_ERROR
 • BROWSER_SET_DEFAULT_RESULT
 • BROWSER_IS_USER_DEFAULT
using about:telemetry, but I was unable to confirm/find the rest of them.

Jared, could you please expand a bit on how to verify all the probes mentioned in Comment 5?

Comment 33

[Jared Wein [:jaws] (please needinfo? me)]

 "BROWSER_IS_USER_DEFAULT_ERROR"

This will only show up if an error is encountered when asking the OS if the browser is default. I don't know a reproducible way to test this, we will just have to monitor the Telemetry dashboard. Can you easily reproduce this one and get it to increment?

 "BROWSER_SET_DEFAULT_DIALOG_PROMPT_RAWCOUNT"

This will show up when the "Set Firefox as Default?" dialog appears. It tracks how many times that dialog shows up. If you saw the dialog you should see this probe in about:telemetry.

 "BROWSER_SET_DEFAULT_ALWAYS_CHECK"

This will show up after the "Set Firefox as Default?" dialog appears and stores the value of the checkbox for "Always Ask".

 "BROWSER_SET_DEFAULT_ERROR"

This will show up if an error in the OS is encountered when trying to set Firefox as the default browser. I don't know a reproducible way to test this, we will just have to monitor the Telemetry dashboard.

 "BROWSER_SET_DEFAULT_TIME_TO_COMPLETION_SECONDS"

This will show up after selecting "Set Firefox as default browser". It records how long it takes to set it as the default browser.

For most of these probes you may want to use a new profile or make sure that the `browser.shell.checkDefaultBrowser` set to true.

Comment 34

[Jared Wein [:jaws] (please needinfo? me)]

I looked back at the patch that landed and found it was missing the declaration for the BROWSER_SET_DEFAULT_ALWAYS_CHECK histogram, which was also causing the BROWSER_SET_DEFAULT_DIALOG_PROMPT_RAWCOUNT histogram to not get recorded.

I still need to look more in to why BROWSER_SET_DEFAULT_TIME_TO_COMPLETION_SECONDS is not showing up on telmetry.mozilla.org.

Comment 35

[Vasilica Tamas, QA [:vtamas]]

There are different behaviours between Firefox versions, so I've noticed the following probes:


[1] In Firefox 41 RC: - BROWSER_IS_USER_DEFAULT
  	              - BROWSER_IS_USER_DEFAULT_ERROR
                      - BROWSER_SET_DEFAULT_RESULT
                      - BROWSER_SET_DEFAULT_ERROR (only when Firefox is set as default browser)

[2] In Firefox 42.0a2:- BROWSER_IS_USER_DEFAULT
                      - BROWSER_IS_USER_DEFAULT_ERROR
                      - BROWSER_SET_DEFAULT_ALWAYS_CHECK
                      - BROWSER_SET_DEFAULT_RESULT
                      - BROWSER_SET_DEFAULT_ERROR (only when Firefox is set as default browser)
                      - BROWSER_SET_DEFAULT_TIME_TO_COMPLETION_SECONDS (only when Firefox is set as default browser)

[3] In Firefox 43.0a1: encountered Bug 1206713 and could not be tested.



Based on the above mentioned and on Bug 1205038, I am marking this bug as Verified fixed on Firefox 41 and Firefox 42.

Comment 36

[Georg Fritzsche [:gfritzsche]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #5)
> The attached patch adds new probes that are opt-out on release channel.
> 
> These are the following probes:
>  "BROWSER_IS_USER_DEFAULT_ERROR": {
>    "description": "True if the browser was unable to determine if the
> browser was set as default."
>  },
...
> And makes the following probe opt-out, to match the probes added by this bug:
>    "BROWSER_IS_USER_DEFAULT": {
>      "description": "The result of the startup default desktop browser
> check."
>    },

I only saw this by accident now, but we already have settings.isDefaultBrowser in the environment, which covers both of these as opt-out.
We shouldn't need either of these two in the release data.

Comment 37

[Jared Wein [:jaws] (please needinfo? me)]

(In reply to Georg Fritzsche [:gfritzsche] from comment #36)
> I only saw this by accident now, but we already have
> settings.isDefaultBrowser in the environment, which covers both of these as
> opt-out.
> We shouldn't need either of these two in the release data.

We are also lacking a simple way of viewing and understanding the data that is captured by the Telemetry Environment. Can you help me set that up?

Comment 38

[Georg Fritzsche [:gfritzsche]]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #37)
> (In reply to Georg Fritzsche [:gfritzsche] from comment #36)
> > I only saw this by accident now, but we already have
> > settings.isDefaultBrowser in the environment, which covers both of these as
> > opt-out.
> > We shouldn't need either of these two in the release data.
> 
> We are also lacking a simple way of viewing and understanding the data that
> is captured by the Telemetry Environment. Can you help me set that up?

Per IRC we should have a follow-up bug for these specific entries.
I'm hoping there is a relatively easy path to make certain well-known paths showing in the dashboard.

Comment 39

[Jared Wein [:jaws] (please needinfo? me)]

See bug 1211583 for dashboard follow-up.

Comment 40

[Vasilica Tamas, QA [:vtamas]]

Tested this bug on Firefox 45.0a1 (2015-11-17), Firefox 44.0a2 (2015-11-17) and Firefox 43 beta 4 under Windows 10 64-bit, Mac OS X 10.11.1 and Ubuntu 12.04 32-bit.

Filed 3 follow-up bugs: Bug 1225098 , Bug 1225780 and Bug 1225814

Based on the above mentions, I am marking this bug as Verified since the other issues are tracked separately.