1432897 - Track Policy Engine status on telemetry

Status: VERIFIED FIXED

(Firefox :: Enterprise Policies, enhancement, P3)

Description

[:Felipe Gomes (needinfo for replies!)]

We should store Services.policies.status somewhere on Telemetry.

This is a numeric field that can have the following values: https://searchfox.org/mozilla-central/source/toolkit/components/enterprisepolicies/nsIEnterprisePolicies.idl#8

Chutten, what do you suggest these days for this?  Telemetry environment, a histogram, something else?

Comment 1

[Chris H-C :chutten]

Do you want it to cause a subsession split when it changes? If not, you don't want it in the environment. Unless you need it to be reported on each and every ping. Is it a pref?

If you don't need it on every ping, it could be a string scalar or a categorical histogram... both of those allow for adding labels/values if needed, later.

Comment 2

[:Felipe Gomes (needinfo for replies!)]

(In reply to Chris H-C :chutten from comment #1)
> If you don't need it on every ping

What's the opposite of "being reported on every ping"? I have been a bit out of touch lately with telemetry.

FWIW this value is only defined on startup so it won't ever change after the beginning of a session

I remember the kind of FLAG histograms that only need to run if a condition is true, so I think that would be ideal for it (only setting it if the engine is actually active).  But I don't know if there's something newer and better than that nowadays

Comment 3

[Chris H-C :chutten]

Not so much opposite as alternative :)

Most of the data we collect you only really care about once. You want to know the max number of tabs someone had open over their session, so we send that once (at the end of the session). You want to know how many tabs were restored in the session restore, so we send that once (near the beginning of the session).

This is why scalars and histograms only appear in pings from subsessions in which they were recorded. So if you have a session with 5 subsessions, the max number of tabs will only be in main ping #5, and the number of tabs restored will only be in main ping #1.

This is the alternative to being in every ping. It makes certain questions harder to answer, but it saves the resources of users and mozilla.

What questions are you hoping to answer with this data?

Comment 4

[:Felipe Gomes (needinfo for replies!)]

Basically this information won't be super useful where we really need it, which is ESR (where many people will have telemetry disabled).

So the information that I _do_ want to answer is: verifying that not a lot of people is using this in the regular release channel.

Comment 5

[Chris H-C :chutten]

For that you can put it in a scalar and then APPROX_DISTINCT(client_id) WHERE that scalar is set to the value you're interested in. Mark it opt-out, give it a tight expiry version (unless you want to monitor it over time. Not sure if you want to verify its deployment (short-term) or its lasting impact (long-term)), then fire up the Data Collection Review process.

Comment 6

[:Felipe Gomes (needinfo for replies!)]

not part of MVP, since this information is not very useful for ESR, as mentioned in comment 4.

Still would be nice to add this at some point for the release channel

Comment 7

[Mike Kaply [:mkaply]]

Attachment: Bug 1432897 - Add count for number of policies to telemetry.

Comment 8

[Mike Kaply [:mkaply]]

Attachment: Data review request

Comment 9

[Chris H-C :chutten]

Comment on attachment 9111749 [details]
Data review request

Load-balancing to Nicole.

Comment 10

[Nicole Shadowen]

Comment on attachment 9111749 [details]
Data review request

Hi, I'd like to complete this data review. Before doing so, can someone please confirm that comments from :chutten have been reviewed and/or responded to in the Scalars.yaml documentation?

Comment 11

[Mike Kaply [:mkaply]]

Yes, I updated the documentation per his request. I just need to correct a typo (double at).

You can see the update here:

https://phabricator.services.mozilla.com/D54799

Comment 12

[Nicole Shadowen]

Comment on attachment 9111749 [details]
Data review request

DATA COLLECTION REVIEW RESPONSE:
Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?
     Yes. This collection is Telemetry so is documented in its definitions file [Scalars.yaml](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/telemetry/Scalars.yaml) and the [Probe    
    Dictionary](https://telemetry.mozilla.org/probe-dictionary/).
Is there a control mechanism that allows the user to turn the data collection on and off?
     Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.
If the request is for permanent data collection, is there someone who will monitor the data over time?
     Yes, mkaply is responsible.
Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?
     Category 2, Interaction. Measurement description: Number of active policies
Is the data collection request for default-on or default-off?
     Default on for all channels.
Does the instrumentation include the addition of any new identifiers?
     No.
Is the data collection covered by the existing Firefox privacy notice?
     Yes.
Does there need to be a check-in in the future to determine whether to renew the data?
     No. This collection is permanent.
Does the data collection use a third-party collection tool?
     No, telemetry only.
---
Result: datareview+

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f4269afef555
Add count for number of policies to telemetry. r=mconley

Comment 14

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/f4269afef555

Comment 15

[Mike Kaply [:mkaply]]

Comment on attachment 9111725 [details]
Bug 1432897 - Add count for number of policies to telemetry.

Beta/Release Uplift Approval Request

• User impact if declined: No user impact, impacts our data collection.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Add a policies.json with a policy. Search on policies in telemetry and verify it's there.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Just adds a telemetry hook in policy
• String changes made/needed:

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: We're looking to get more data on policy usage, and ESR is where policies are used the most.
• User impact if declined: No user impact, impacts our data collection.
• Fix Landed on Version: 74
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Just adds a telemetry hook in policy
• String or UUID changes made by this patch:

Comment 16

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9111725 [details]
Bug 1432897 - Add count for number of policies to telemetry.

Adds telemetry for policy engine usage. Approved for 73.0b4. Will revisit ESR after this has baked on Beta for a bit.

Comment 17

[Dorel Luca [:dluca]]

https://hg.mozilla.org/releases/mozilla-beta/rev/69d8d30152ea

Comment 18

[Ciprian Georgiu, Desktop QA]

I have verified that the policies added in policies.json are correctly counted in about:telemetry#scalars-tab. This was tested on Win 10 x64, macOS 10.15 and Ubuntu 18.04 running Firefox Beta 73.0b4 and Nightly 74.0a1.

Comment 19

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9111725 [details]
Bug 1432897 - Add count for number of policies to telemetry.

Thanks for the verification. Approved for 68.5esr also.

Comment 20

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/1d4968094d07

Comment 21

[Ciprian Georgiu, Desktop QA]

I have verified this implementation on 68.5.0esr as well, and I can confirm that it's working as expected. Tested on Win 10 x64, macOS 10.13 and Ubuntu 18.04 x64.