1412382 - Improve display of use counters

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Georg Fritzsche [:gfritzsche]]

Per bug 1409865, use counters should probably be always interpreted relative to TOP_LEVEL_CONTENT_DOCUMENTS_DESTROYED & CONTENT_DOCUMENTS_DESTROYED:
https://searchfox.org/mozilla-central/rev/dd47bee6468de7e1221b4d006342ad6b9813d0e5/dom/base/nsDocument.cpp#13169

I put a temporary hacky dashboard up here:
https://georgf.github.io/usecounters/index.html

We should think about how to properly deal with use counters in the future.
Should they have specialized views in the TMO dashboard?
Or should they just have their own dashboard?

Comment 1

[Georg Fritzsche [:gfritzsche]]

This will need a bit of understanding of the future of the TMO dashboards.

Comment 3

[Georg Fritzsche [:gfritzsche]]

So, digging a bit with others, it turns out that in follow-up to bug 1204994, the aggregator is doing some processing of USE_COUNTERS2_*:
https://github.com/mozilla/python_mozaggregator/blob/master/mozaggregator/aggregator.py#L190

AFAICT, this does the following:
(1) we leave the total value in the "1" bucket
(2) we add a normalized value in the "0" bucket [1]: (total - 1-bucket-sum)

This mean that we might show correct values on the TMO dashboard.
It is confusing though, as this context is hard to understand.

@frank,chutten: does that sound correct?

@chutten: with the information we have now, does my dashboard [2] look correct?
Should this https://mzl.la/2zldel0 match up with [3]?

[1] relative to TOP_LEVEL_CONTENT_DOCUMENTS_DESTROYED / CONTENT_DOCUMENTS_DESTROYED.
[2] https://georgf.github.io/usecounters/index.html
[3] https://georgf.github.io/usecounters/index.html#kind=page&group=JS

Comment 4

[Chris H-C :chutten]

The histogram as accumulated on the client only puts values into the 1 bucket. Then, for any client with at least one sample in the 1 bucket, the histogram is filled in by the aggregator with all of the 0s it should have due to pages it visited that didn't set a 1.

This means we don't aggregate pings which, during their subsessions, don't see a counted feature. We don't insert use counters full of 0s on the aggregator side (at least not that I have found).

This means TMO provides an accurate picture of how often a feature is used within sessions that have used it. This isn't useful in figuring out if a feature is used across the web, but it might be a better measure to guess how much users who _do_ see the feature actually care. This is a bit flimsy, but is similar to what we did for hangs for the longest time: disregard users who don't experience them.

This also means your dashboard provides an accurate picture of how often a feature is used across all sessions you've examined. This should provide a more useful value for how often a feature is "seen" throughout the portion of the Web experienced by the users reporting this data.

Your numbers should not line up with TMO's unless, for the matching use counter, every subsession saw at least one use of the feature.

Comment 5

[Luke Wagner [:luke]]

I expect casual users of usage counters are expecting they are seeing a total % of page/document loads, not "of subsessions that use the feature at least once"; I was and that's what the docs seem to say:
  https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/collection/use-counters.html
It seems like a bug to report anything else.

Comment 6

[Frank Bertsch [:frank]]

Current explanations look correct.

We could display % of page/document loads in TMO by adding some front-end only work that takes the counter and TOP_LEVEL_CONTENT_DOCUMENTS_DESTROYED or CONTENT_DOCUMENTS_DESTROYED, and displays the ratio. In that case, it would just be "SUM(number of elements in the 1 bucket) / SUM(total)" for the selected dimensions.

Comment 7

[Chris H-C :chutten]

So the plan is: use the true values from USE_COUNTER2* probes and normalize against the count of (TOP_LEVEL_)CONTENT_DOCUMENTS_DESTROYED in the pie charts instead of using the false values, in TMO's front-end.

No changes planned for the aggregator, though I suppose we'll be able to ditch the existing use-counter-related shenanigans if we want to.

Comment 8

[Georg Fritzsche [:gfritzsche]]

We should probably also inform users of the fact that these are proportional top-level page / document.
Possibly in a follow-up?
Use counters don't have a description now, is that a place to put context?

Comment 9

[Chris H-C :chutten]

Fixed at the service level: https://github.com/mozilla/python_mozaggregator/pull/59

Email sent to dev-platform informing of the problem, now that it's fixed :S https://groups.google.com/forum/#!topic/mozilla.dev.platform/3pdSQtwOz5g

Blogpost written: https://chuttenblog.wordpress.com/2018/01/17/firefox-telemetry-use-counters-over-estimating-usage-now-fixed/