Closed Bug 1461439 Opened 3 years ago Closed 3 years ago

Collect browser error telemetry metrics in beta and release

Categories

(Firefox :: General, enhancement)

enhancement
Not set
normal

Tracking

()

RESOLVED FIXED
Firefox 62
Tracking Status
firefox62 --- fixed

People

(Reporter: osmose, Assigned: osmose)

References

Details

Attachments

(2 files)

In bug 1444554, we added some telemetry scalars for measuring the number of browser chrome JS errors occurring and being collected by our error reporting prototype. All browser error collection, including collecting these scalars, is currently limited to the Nightly channel.

Of the scalars added in that bug, I'd like to enable collection of 3 of them in beta and release builds:

browser.errors.collected_count
    The count of all browser chrome JS errors that were collected locally.

browser.errors.collected_with_stack_count
    The count of browser chrome JS errors that were collected locally and had
    a usable stack trace.

browser.errors.collected_count_by_filename
    The count of all browser chrome JS errors that were collected locally,
    keyed by the filename of the file in which the error occurred.

The motivation for this is that I'd like to use the data to look for correlations between the number of JS errors a user experiences, and quality metrics, such as usage hours, satisfaction, and retention. We'll need a backlog of data in order to get reliable results from this, so I'm hoping to land this collection early before we can get some time from a data scientist to help write the analysis code.

Reporting errors to Sentry should stay fully disabled outside of the Nightly channel. Only the aggregate telemetry counts should be enabled. I don't think we need to change any privacy notices as long as that stays disabled.
I missed clarifying in the summary, but I'm planning on including the about:preferences pref for error collection in this to let users opt-out of this collection specifically. Collecting errors themselves will still be hard-disabled on non-Nightly builds regardless of the pref value.
Attachment #8975621 - Flags: review?(francois)
(In reply to Michael Kelly [:mkelly,:Osmose] from comment #1)
> I missed clarifying in the summary, but I'm planning on including the
> about:preferences pref for error collection in this to let users opt-out of
> this collection specifically. Collecting errors themselves will still be
> hard-disabled on non-Nightly builds regardless of the pref value.

You're talking about "Allow Nightly to send browser error reports (including error messages) to Mozilla" ?

If I understand correctly, on Nightly, that pref:

- disables the sending of these scalars
- disables the sending of the actual error messages

and then on beta/release that pref:

- disables the sending of these scalars
- (sending of actual error messages is always disabled)

Is that right?
Flags: needinfo?(mkelly)
(In reply to François Marier [:francois] from comment #2)
> You're talking about "Allow Nightly to send browser error reports (including
> error messages) to Mozilla" ?

I am a victim of my own hubris. I went off memory instead of actually checking the wording, that's totally not generic like I said it was. My bad.

> If I understand correctly, on Nightly, that pref:
> 
> - disables the sending of these scalars
> - disables the sending of the actual error messages
> 
> and then on beta/release that pref:
> 
> - disables the sending of these scalars
> - (sending of actual error messages is always disabled)
> 
> Is that right?

That was my intent, yes.

I could change the wording on release and beta to not include the "(including error messages)" bit if that'd be better. I was just trying to avoid making extra l10n work if we could avoid it. Not bothering with a specific opt-out for this data, or making it a non-user-visible pref, are also fine options from my viewpoint.
Flags: needinfo?(mkelly)
(In reply to Michael Kelly [:mkelly,:Osmose] from comment #3)
> Not bothering with a specific opt-out for this data, or making it a non-user-visible pref, are
> also fine options from my viewpoint.

Since users who care can already opt out by disabling telemetry, that's probably enough. I don't think you need to have a specific opt-out for just these scalars.

Can you remind me where the documentation is publicly posted for this collection? I need this to answer Q1 on https://github.com/mozilla/data-review/blob/master/review.md.
Flags: needinfo?(mkelly)
(In reply to François Marier [:francois] from comment #4)
> Can you remind me where the documentation is publicly posted for this
> collection? I need this to answer Q1 on
> https://github.com/mozilla/data-review/blob/master/review.md.

Just to clarify, if it's all in Scalars.yaml, that's fine. I just vaguely remember that there might be other docs somewhere about this feature.
(In reply to François Marier [:francois] from comment #4)
> (In reply to Michael Kelly [:mkelly,:Osmose] from comment #3)
> > Not bothering with a specific opt-out for this data, or making it a non-user-visible pref, are
> > also fine options from my viewpoint.
> 
> Since users who care can already opt out by disabling telemetry, that's
> probably enough. I don't think you need to have a specific opt-out for just
> these scalars.
> 
> Can you remind me where the documentation is publicly posted for this
> collection? I need this to answer Q1 on
> https://github.com/mozilla/data-review/blob/master/review.md.

Scalars.yml is all there is for the telemetry data. The collection of error data itself is documented here: https://firefox-source-docs.mozilla.org/browser/browser/BrowserErrorReporter.html

I wasn't planning on updating that since it doesn't cover the scalars at all, but I could if you think it's worth mentioning there.
Flags: needinfo?(mkelly)
Comment on attachment 8975621 [details]
Data Collection Request

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, Scalars.yaml.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Not permanent.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 1.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

Yes, when the experiment completes.
Attachment #8975621 - Flags: review?(francois) → review+
We probably want to wait until the patch for idle processing lands before enabling this in beta/release for perf reasons.
Depends on: 1464521
Comment on attachment 8985716 [details]
Bug 1461439: Enable browser error telemetry on non-Nightly channels.

https://reviewboard.mozilla.org/r/251260/#review257684

::: browser/modules/test/browser/browser_BrowserErrorReporter_nightly.js:1
(Diff revision 1)
> +/* Any copyright is dedicated to the Public Domain.

Can you preserve blame by marking this as a copy of the older file, perhaps using something like:

```
hg histedit
# edit this commit
cp browser/modules/test/browser/browser_BrowserErrorReporter_nightly{,2}.js
hg cp --after --force browser/modules/test/browser/browser_BrowserErrorReporter{,_nightly}.js
mv browser/modules/test/browser/browser_BrowserErrorReporter_nightly{2,}.js
hg histedit --continue
```

please?
Attachment #8985716 - Flags: review?(gijskruitbosch+bugs) → review+
Pushed by mkelly@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a538fcc4c124
Enable browser error telemetry on non-Nightly channels. r=Gijs
https://hg.mozilla.org/mozilla-central/rev/a538fcc4c124
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 62
You need to log in before you can comment on or make changes to this bug.