1530508 - Add more telemetry data around Normandy filters

Status: RESOLVED FIXED

(Firefox :: Normandy Client, enhancement)

Description

[Ethan Glasser-Camp (:glasserc)]

The Normandy system is a complicated one with many components. Recipes/experiments are adjusted on a server; those recipes and experiments are delivered to the browser; client code decides which recipes to execute, and then executes them; those recipes hopefully change the behavior of Firefox; eventually, these recipes end and the client unwinds them. Because of the complexity of the system and its role in performing experiments, it can be hard to "see inside" and make sure things are working the way we intended and that the results we get are valid.

We already have some uptake telemetry around Normandy, which gives us some visibility. However, there are a few questions that are hard to answer today.

• Is a given recipe being considered by clients, and what ratio of our clients are choosing to apply it? -- Being able to answer this question allows our data scientists to ensure they have the right level of statistical power to estimate the significance of their results. Right now this question is hard to answer because we only report the outcome of recipes that we receive AND decide to apply, and we don't have a way to answer:
• Is Normandy client getting the right recipes? We've never had a problem with this per se, but it presents a confounding factor above and it serves as a valuable health check to have confidence in this system's behavior going forward.
• Is there any problem with the filter? We've had some bugs (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1477156) where a filter was itself invalid. Most things in Normandy already report errors, but this particular code path isn't instrumented.

To address this bug, I would like to do the following:

• Add uptake telemetry to report a status for recipes that we consider and do not match the filter for. Because we already report uptake telemetry for recipes that we consider and do match the filter for, this will solve #1, allowing us to calculate the ratio of clients that are choosing to apply it. Additionally, this is the main "hole" in uptake telemetry for recipes so it would allow us to get some insight into point #2, are we getting the right recipes.
• Add a keyed histogram of recipe ID and recipe last_modified. This might be excessively paranoid but it helps ensure that #2 is addressed, because we will be recording not only the recipe IDs but also gives us information indicating the freshness of that recipe.
• Wrap the Jexl filter execution in a try/catch and log in uptake telemetry a CONTENT_ERROR for the recipe if the Jexl filtering threw an exception.

Comment 1

[Ethan Glasser-Camp (:glasserc)]

Attachment: data-collection.txt

Hi :chutten, I know you have a little bit of the context here but feel free to pass review to someone better suited to it.

Comment 2

[Chris H-C :chutten]

Comment on attachment 9046786 [details]
data-collection.txt

DATA COLLECTION REVIEW RESPONSE:

    Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes. This collection is Telemetry so is documented in its definitions file and the [Probe Dictionary](https://telemetry.mozilla.org/probe-dictionary/).

    Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

    If the request is for permanent data collection, is there someone who will monitor the data over time?

Yes. :glasserc is responsible.

    Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

    Is the data collection request for default-on or default-off?

Default on for all channels.

    Does the instrumentation include the addition of any new identifiers?

No. (the normandy recipe id is a known identifier)

    Is the data collection covered by the existing Firefox privacy notice?

Yes.

    Does there need to be a check-in in the future to determine whether to renew the data?

No. This collection is permanent.

---
Result: datareview+

Comment 3

[Ethan Glasser-Camp (:glasserc)]

Attachment: Bug 1530508: convert reportRecipe to take the recipe, not just its ID

This will make it easier to report recipe freshness.

Comment 4

[Ethan Glasser-Camp (:glasserc)]

Attachment: Bug 1530508: report freshness of each recipe

Depends on D22016

Comment 5

[Ethan Glasser-Camp (:glasserc)]

Attachment: Bug 1530508: report additional recipe information

Report when recipes don't match the filter. Report when Jexl filters
themselves fail, with an added test.

The existing test for remote-settings usage had a bug, so fix that
too.

Depends on D22017

Comment 6

[Ethan Glasser-Camp (:glasserc)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=a1452cfc91851dbdf20485ab15720af32abb1a0e showed a bug, so I'm starting https://treeherder.mozilla.org/#/jobs?repo=try&revision=d8ee443fda4cbb316ee313cbb959284df30aedc2 .

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fa9146245803
convert reportRecipe to take the recipe, not just its ID r=mythmon
https://hg.mozilla.org/integration/autoland/rev/0d1b3156e7c8
report freshness of each recipe r=mythmon
https://hg.mozilla.org/integration/autoland/rev/10333e6aa4b0
report additional recipe information r=mythmon

Comment 8

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/fa9146245803
https://hg.mozilla.org/mozilla-central/rev/0d1b3156e7c8
https://hg.mozilla.org/mozilla-central/rev/10333e6aa4b0