1569672 - Collect denominators for use counters in release

Status: RESOLVED FIXED

(Toolkit :: Telemetry, task)

Description

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Use counters encode the count of documents that have used a given platform feature. The CONTENT_DOCUMENTS_DESTROYED and TOP_LEVEL_CONTENT_DOCUMENTS_DESTROYED probes, as implemented in bug 1204994, are the denominators for the use counters.

Bug 1477433 began collecting use counters in release, but the denominators are still only recorded in prerelease. We need to flag the denominator histograms for collection in release.

Comment 1

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Attachment: Bug 1569672 - Collect use counter denominators in release

Use counters are collected in release as of bug 1477433. Per bug 1204994, use counters need to be interpreted relative to CONTENT_DOCUMENTS_DESTROYED and TOP_LEVEL_CONTENT_DOCUMENTS_DESTROYED, so these need to be collected in release, as well.

Also cleans up probe definitions and removes from whitelists.

Comment 2

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=518c0061ad180d6cb82808d00e0e107563dff3d9

Comment 3

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

try https://treeherder.mozilla.org/#/jobs?repo=try&revision=b75b33f7221a468a6041e34581ccb9fc15707c72

Comment 4

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Attachment: Data collection review

Comment 5

[Ben Miroglio [:bmiroglio]]

Comment on attachment 9081364 [details]
Data collection review


1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way? 
Data will be documented in the [probe dictionary](https://telemetry.mozilla.org/probe-dictionary/). 

2. Is there a control mechanism that allows the user to turn the data collection on and off? 
Yes, can be turned off in preferences. 

3. If the request is for permanent data collection, is there someone who will monitor the data over time?
Yes, tdsmith + the client telemetry team

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
Category 2 - Interaction Data

5. Is the data collection request for default-on or default-off?
Default on. 

6. Does the instrumentation include the addition of any new identifiers?
No new identifiers. 

7. Is the data collection covered by the existing Firefox privacy notice?
Yes.

8. Does there need to be a check-in in the future to determine whether to renew the data? 
No. 

9. Does the data collection use a third-party collection tool? 
No. 
—
Data review +

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/254df321d0df
Collect use counter denominators in release r=janerik

Comment 7

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/254df321d0df

Comment 8

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9081336 [details]
Bug 1569672 - Collect use counter denominators in release

Beta/Release Uplift Approval Request

• User impact if declined: These probes make it possible to interpret the use counters that have shipped in release. Having these probes available will help us make better webcompat decisions sooner.
• Is this code covered by automated tests?: Unknown
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Not risky; change limited to histogram definitions.
• String changes made/needed: none

Comment 9

[Mike Taylor [:miketaylr]]

Comment on attachment 9081336 [details]
Bug 1569672 - Collect use counter denominators in release

If possible, getting this out in the next dot release would be super helpful for current webcompat metrics efforts.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9081336 [details]
Bug 1569672 - Collect use counter denominators in release

Improvements to use counters so we can better interpret the data coming from them. Approved for 69.0b10.

I'm less enthusiastic about uplifting to release for 68.0.2, however. This doesn't appear to be fixing any user-facing issue or a big regression shipped. What webcompat decisions are going to be made easier by getting newer data from this probe a month before we ship Fx69?

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/86fb2666395c

Comment 12

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

I'm less enthusiastic about uplifting to release for 68.0.2, however. This doesn't appear to be fixing any user-facing issue or a big regression shipped. What webcompat decisions are going to be made easier by getting newer data from this probe a month before we ship Fx69?

I'm working with Mike (who's on PTO for the next few days) to produce metrics for Firefox webcompat in support of the Gecko codevelopment goal. The specific things I want to do soon are a) to compare numbers coming out of our use counter infrastructure with Chrome's use counter data to understand whether they're comparable, and b) to compare use counter numbers with crawls that we're going to start piloting this month to understand whether they're comparable. Ultimately, we want to count attempts by web pages to use APIs that we don't support, and we think that a use-counter-like mechanism will be the way to go. If our use counters indicate something radically different from the other datasets, we'll want to figure out why. Waiting for beta (for which: thanks!) won't kill us, but riding along would help remove some risk from our schedule, which is targeting late September for our key deliverables.

Comment 13

[Julien Cristau [:jcristau]]

How do bugs like 1568950 affect that data and its analysis?

Comment 14

[Julien Cristau [:jcristau]]

Comment on attachment 9081336 [details]
Bug 1569672 - Collect use counter denominators in release

anyway.. I guess we can take this as a ride along, assuming there are no perf implications to enabling that collection on release. approved for 68.0.2.

Comment 15

[Julien Cristau [:jcristau]]

https://hg.mozilla.org/releases/mozilla-release/rev/7ece03f69719