1295588 - Add telemetry to track which events document.createEvent is used for

Status: RESOLVED FIXED

(Core :: DOM: Events, defect)

Description

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

We want to spec which values are needed for web-compat, so let's see what's actually compatible for us.

Comment 1

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

Attachment: Patch

I don't know what the policy is on making telemetry opt-out, so please let me know if I should remove that and make them opt-in.  My reasoning was that 1) the information disclosure from these flags seems minimal, and 2) it's possible that some events are used by specific programs that may be underrepresented in the opt-in population, so ideally we should have a fuller picture.  But if they should be opt-in, then that should be perfectly serviceable too.

Comment 2

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

Also, I didn't provide a contact e-mail, because I don't think it's useful to know if these values change.  We'll want them for as long as it takes to reach consensus among UAs about which values to keep in the spec, then we'll probably want to remove them.  This will take an indeterminate amount of time, so I didn't put an expiration date on them, but fluctuations in usage within the period they exist are not interesting.

Comment 3

[Olli Pettay [:smaug][[e-mail]]]

Comment on attachment 8782410 [details] [diff] [review]
Patch

Not sure why expire is never, but perhaps that doesn't really matter either.
But bsmedberg needs to review/feedback+ this too.

Comment 4

[Benjamin Smedberg]

Comment on attachment 8782410 [details] [diff] [review]
Patch

opt-out is fine, but you should be aware that you'll have to write your own analysis, because telemetry.mozilla.org only reports from the prerelease/opt-in population.

It seems like the need here is for a specific time-limited purpose. In that case, you should collect the data for a short amount of time (6 months/5 releases is most common). If you need the data longer, I need you to provide more details about what questions long-term data collection answers and what kind of reporting/alerting system you're going to build for it.

"flag" histograms like this can be more expensive to compute, because (AIUI) we include a value in every payload. I suggest that it would be better to make this a count histogram which will then only report when each of these is actually used.

data-review=me with those two changes (count and expiry in Fx56), or come back to me if you need something else.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/f0f12701aeb7
Add createEvent() telemetry; r=smaug, data-review=bsmedberg

Comment 6

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

I wound up making it opt-in, because I don't want to write my own analysis.  I also added my e-mail address as the contact, because it wouldn't let me add new probes otherwise.

(Surely flags should also be sent only if true?)

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/f0f12701aeb7

Comment 8

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

I see the data, but how do I figure out what percentage of data submissions hit it at least once?  E.g.:

https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=0&end_date=2016-08-22&keys=__none__!__none__!__none__&max_channel_version=nightly%252F51&measure=CREATE_EVENT_EVENT&min_channel_version=null&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-08-22&table=0&trim=1&use_submission_date=0

I see the total number of data points received, but not how many data submissions did *not* hit the telemetry.  Is there some way to figure this out, or do I really need flag instead of count here after all?  (If the number of bits being sent is a problem, it would be fine to have this on for one release, or probably even less.)

How am I meant to understand the numbers anyway?  If something hit the accumulator 125 times, does that mean 125 times in one day, one session, one page view, or something else?

Comment 9

[Benjamin Smedberg]

I don't think you've thought through your question very precisely. Could you take a minute to rephrase the question as precisely as possible?

For example, do you care about what percent of users hit a certain event? Do you care how often during browsing an average user hits certain kinds of events? Or do you just care about whether some values were used *ever*? What decision are you going to make as a result of this data? I don't really understand how this data is going to affect standardization decisions.

Your questions in comment 8 are phrased in terms of "data submissions", which is pretty much meaningless: telemetry is broken up into chunks based on the Firefox session, but also 24-hour boundaries and other circumstances.

The chances of the default telemetry.mozilla.org dashboard answering any of these questions is pretty much nil. You're going to need to answer the actual questions with a custom-built query: I am happy to introduce you to the data engineers who can provide examples, assistance, and mentoring.

Comment 10

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

Nathan, is there any way to get "percentage of page views that hit this line" for things like this?  Bug 968923 seems to have only intended to add statistics for access to a property or method, but I don't see whether there's a way to get the same data for something like this (percentage of page views that call a method with a given parameter).  If there's no way to do it now, I'll file a bug.

Comment 11

[Nathan Froyd [:froydnj]]

(In reply to :Aryeh Gregor (working until September 2) from comment #10)
> Nathan, is there any way to get "percentage of page views that hit this
> line" for things like this?  Bug 968923 seems to have only intended to add
> statistics for access to a property or method, but I don't see whether
> there's a way to get the same data for something like this (percentage of
> page views that call a method with a given parameter).  If there's no way to
> do it now, I'll file a bug.

We don't have a way to do that.

Comment 12

[Aryeh Gregor (:ayg) (no longer with Mozilla)]

Filed bug 1297457.