1602912 - Add support for graduation lists

Status: RESOLVED FIXED

(Firefox :: Normandy Client, enhancement, P2)

Description

[Michael Cooper [:mythmon]]

Currently, preference rollouts graduate when the built-in pref matches the rollout pref. Add-on rollouts do not have a graduation mechanism, but it is planned that they would graduate if the add-on is added in-tree. This is a good default, but does not cover all cases.

Sometimes the pref is removed instead of being changed. Sometimes the pref cannot be made built-in, since it relies on runtime information (like country). Sometimes instead of graduating to a built-in system add-on, add-ons become non-addon features of Firefox.

To summarize, the automatic features of graduation are good, but not enough. We should have a second, more flexible method.

I propose the concept of a graduation list. This would be a simple array of rollout slugs. If a rollout is active that has a slug in this list, it will be immediately graduated. Rollouts that are already in this list will be reported as enrollment failures, in the same vein as would-be-noop.

This list does not need to be stored in Remote Settings, and can be a static set of strings in the source. The graduation list should be used in conjunction with other code changes in the tree that make a rollout obsolete. It should not be used as an off-tree way to end a rollout, or as a way to make changes on certain dates. That's what rollbacks are for.

Comment 1

[Michael Cooper [:mythmon]]

Attachment: Bug 1602912 - Add Normandy graduation set r=gijs

Comment 2

[Michael Cooper [:mythmon]]

Attachment: data-request-1675388.md

Hi Tim, since I'm adding a little more telemetry here, Gijs asked that I request data review. The new data is nearly trivial, and I don't expect there is any problem with it's collection. More interestingly though, I wanted to check that adding new extra keys to event types doesn't have any problems as I've done it here.

Comment 3

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9199868 [details]
data-request-1675388.md

lgtm, thanks!

More interestingly though, I wanted to check that adding new extra keys to event types doesn't have any problems as I've done it here.

Not aware of any concerns; we do it a lot.

1. Is there or will there be documentation that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, in the probe definition files and the Probe Dictionary.

2. Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out.

3. If the request is for permanent data collection, is there someone who will monitor the data over time?

4. Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Cat 1

5. Is the data collection request for default-on or default-off?

Default-on

6. Does the instrumentation include the addition of any new identifiers (whether anonymous or otherwise; e.g., username, random IDs, etc. See the appendix for more details)?

Nope

7. Is the data collection covered by the existing Firefox privacy notice?

Yes

8. Does there need to be a check-in in the future to determine whether to renew the data?

No, permanent collection

9. Does the data collection use a third-party collection tool?

No.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2ef0dbf3ff50
Add Normandy graduation set r=Gijs

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/2ef0dbf3ff50