Closed Bug 1440729 Opened 2 years ago Closed 2 years ago

Track console API usage with telemetry

Categories

(Core :: DOM: Core & HTML, enhancement)

enhancement
Not set

Tracking

()

RESOLVED FIXED
mozilla60
Tracking Status
firefox60 --- fixed

People

(Reporter: bgrins, Assigned: bgrins)

Details

Attachments

(2 files)

There's an ongoing discussion about removing `console.exception` from the spec - see https://groups.google.com/d/msg/mozilla.dev.developer-tools/s-Bkwd79DFE/NdhlzNvVCAAJ and https://github.com/whatwg/console/issues/27#issuecomment-366326423.

I expect relatively low usage given that this method is an alias for the more popular `console.error` and that only Firefox and Edge implement the method. But given potential web compat issues, before agreeing to remove it from the spec I'd like to get some data on usage.

At a minimum I'd like to measure usage of console.error and console.exception, but if the cost isn't too high to add more probes it would be useful to track other API calls (log, dir, table, etc) to help prioritize product decisions for the webconsole.
My gut feeling is that on the scale of the work console methods have to do already the cost of the probes will not even be measurable.
Note: documentation for adding this can be found at https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/use-counters.html#webidl-methods-and-attributes. And other bugs that have added similar tracking can be found in the log of UseCounters.conf (https://hg.mozilla.org/mozilla-central/log/tip/dom/base/UseCounters.conf).
I'm not sure if I've done this right - if I load a page that does a bunch of console API calls (i.e. https://bgrins.github.io/devtools-demos/console/all.html) then open about:telemetry I can't see any reference to "console" (or "USE_COUNTER" for that matter). Is there some way to confirm this is working?
Flags: needinfo?(amarchesini)
Well, first of all UseCounter seems to work on main-thread only and not in workers, but I guess this doesn't really matter here.

Yes, your code looks correct. I also tested locally and SetDocumentAndPageUseCounter() is correctly called. Probably you need to wait before having that data shown in about:telemetry.
Flags: needinfo?(amarchesini)
Comment on attachment 8953685 [details]
Bug 1440729 - Include use counters for the console API

François, is collecting data on console API usage in pages OK? AIUI based on https://wiki.mozilla.org/Firefox/Data_Collection this is Category 1 data.
Attachment #8953685 - Flags: feedback?(francois)
(In reply to Brian Grinstead [:bgrins] from comment #6)
> Comment on attachment 8953685 [details]
> Bug 1440729 - Include use counters for the console API
> 
> François, is collecting data on console API usage in pages OK? AIUI based on
> https://wiki.mozilla.org/Firefox/Data_Collection this is Category 1 data.

If it's an API/feature that websites use, then yes it's generally Category 1.

Could you please fill out the data request form at https://github.com/mozilla/data-review/blob/master/request.md? You can attach it to this bug as a text file and r? me on it.

I will take care of doing the data review (i.e. https://wiki.mozilla.org/Firefox/Data_Collection#Step_2:_Request_is_reviewed).
Attachment #8954979 - Flags: review?(francois)
Comment on attachment 8954979 [details]
console-data-request.txt

1) Is there or will there be **documentation** that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes, in this data review and as part of the telemetry docs.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, telemetry setting.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?**

Not permanent. Brian Grinstead will monitor it for at least one release.

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?  **

Category 1.

5) Is the data collection request for default-on or default-off?

Default on.

6) Does the instrumentation include the addition of **any *new* identifiers** (whether anonymous or otherwise; e.g., username, random IDs, etc.  See the appendix for more details)?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No. Some of these probes might get removed along with the features if they have low usage.
Attachment #8954979 - Flags: review?(francois) → review+
Comment on attachment 8953685 [details]
Bug 1440729 - Include use counters for the console API

Thanks for filling out the data review form Brian.

I'd suggest a DOM peer or WebIDL person for the actual patch.

datareview+
Attachment #8953685 - Flags: feedback?(francois)
Attachment #8953685 - Flags: review?(amarchesini)
Assignee: nobody → bgrinstead
Status: NEW → ASSIGNED
Comment on attachment 8953685 [details]
Bug 1440729 - Include use counters for the console API

https://reviewboard.mozilla.org/r/222896/#review230426

Let me know what you think about this comment. If you really need use-counters for all these methods, r+, otherwise, just keep what you really need.

::: dom/base/UseCounters.conf:118
(Diff revision 1)
> +method console.time
> +method console.timeEnd
> +method console.exception
> +method console.timeStamp
> +method console.profile
> +method console.profileEnd

Why do we need to have use-counters for all these methods if the bug is about console.exception?

I guess there are stable methods such as console.log/error/debug/count/clear...

Maybe we don't need use-conters for them.
Attachment #8953685 - Flags: review?(amarchesini)
(In reply to Andrea Marchesini [:baku] from comment #11)
> Comment on attachment 8953685 [details]
> Bug 1440729 - Include use counters for the console API
> 
> https://reviewboard.mozilla.org/r/222896/#review230426
> 
> Let me know what you think about this comment. If you really need
> use-counters for all these methods, r+, otherwise, just keep what you really
> need.
> 
> ::: dom/base/UseCounters.conf:118
> (Diff revision 1)
> > +method console.time
> > +method console.timeEnd
> > +method console.exception
> > +method console.timeStamp
> > +method console.profile
> > +method console.profileEnd
> 
> Why do we need to have use-counters for all these methods if the bug is
> about console.exception?
> 
> I guess there are stable methods such as
> console.log/error/debug/count/clear...
> 
> Maybe we don't need use-conters for them.

console.exception / console.error is the motivator to land this, but we've been talking about measuring the console API for a long time to help with prioritizing feature work on the console UI, and potentially to remove other methods not currently in the spec like profile/profileEnd. We can retire these counters after we answer those questions. ni? Harald to confirm this is accurate.
Flags: needinfo?(hkirschner)
There are general product questions on what kind of Console messages developers use, so we can make decisions on what messages to focus on for polish. The spec work Brian mentioned is of course also something that want to have data on to support decisions.

What was not discussed here yet is the web compat aspect, which is a sideffect but came up in discussions with the compat team. When we run experiments changing platform behavior, seeing a change in Console usage between branches is an indicator that Firefox started hitting different code paths on applications. While errors are the most interesting number, the distributions of all Console usage puts in a relative context.
Flags: needinfo?(hkirschner)
Comment on attachment 8953685 [details]
Bug 1440729 - Include use counters for the console API

Re-request as per Comments 12/13
Attachment #8953685 - Flags: review?(amarchesini)
Comment on attachment 8953685 [details]
Bug 1440729 - Include use counters for the console API

https://reviewboard.mozilla.org/r/222896/#review232036
Attachment #8953685 - Flags: review?(amarchesini) → review+
Pushed by bgrinstead@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/da5ed9471918
Include use counters for the console API r=baku
https://hg.mozilla.org/mozilla-central/rev/da5ed9471918
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
We're seeing 0 usage of console.exception on beta from these probes (https://georgf.github.io/usecounters/index.html#kind=page&group=CONSOLE&channel=beta&version=60).

Mike, do you have an opinion as to whether we should remove our implementation and remove it from the spec, or keep it around as a deprecated method?
Flags: needinfo?(miket)
0.000% is a pretty compelling number.... no objections from me for removal.
Flags: needinfo?(miket) → needinfo?(bgrinstead)
(In reply to Mike Taylor [:miketaylr] from comment #20)
> 0.000% is a pretty compelling number.... no objections from me for removal.

I just looked up 61 nightly and I see 0.001%, which I guess at least confirms that we are gathering data on it: https://georgf.github.io/usecounters/index.html#kind=page&group=CONSOLE&channel=nightly&version=61.

Does having virually 0 instead of absolutely 0 usage change your opinion? If not, I will comment in the github issue that we are OK to remove it.
Flags: needinfo?(bgrinstead) → needinfo?(miket)
(In reply to Brian Grinstead [:bgrins] from comment #21)
> Does having virually 0 instead of absolutely 0 usage change your opinion? If
> not, I will comment in the github issue that we are OK to remove it.

I don't think so. If we encounter issues where sites are breaking, we should be able to shim it with our system addon. Let's go ahead and remove.
Flags: needinfo?(miket)
Component: DOM → DOM: Core & HTML
You need to log in before you can comment on or make changes to this bug.