1685134 - unenrollment normandy test with dummy pref

Status: RESOLVED FIXED

(Firefox :: Normandy Client, enhancement, P1)

Description

[:shell escalante]

https://experimenter.services.mozilla.com/experiments/system-test-no-change-boolean/
https://experimenter.services.mozilla.com/experiments/unenrollment-normandy-test-with-no-op-pref-in-firefox-integer/
https://experimenter.services.mozilla.com/experiments/unenrollment-normandy-test-with-no-op-pref-in-firefox-string/

test with boolean, integer, and string prefs - ONLY ON NIGHTLY AND BETA - won't ride to release. Experiment will only use on nightly right now -but we'll leave these prefs in for future nothing burger tests.

https://protosaur.dev/partybal/bug_1676470_pref_warpbuilder_experiment_release_83_83.html#unenroll
https://experimenter.services.mozilla.com/experiments/warpbuilder-experiment/

https://protosaur.dev/partybal/bug_1672866_pref_network_partitioning_release_83_84.html#unenroll
https://experimenter.services.mozilla.com/experiments/network-partitioning/

Comment 1

[Michael Cooper [:mythmon]]

The goal here is to test if the recent elevated unenrollments on one branch or another are caused by feature-specific interference, or is something inherent to Normandy. The idea is to have a few preferences that don't control anything and aren't modified by any UI or automated systems. One preference per preference type (int, bool, string). We can then launch experiments and rollouts against these preferences to test if they behave as expected when we know the full list of influences on the preference.

Comment 2

[Michael Cooper [:mythmon]]

These preferences should only be present on pre-release versions, the should not ride the trains to Release.

Comment 3

[Tif Tran]

Attachment: Bug 1685134 - Added Normandy Do Nothing Prefs for testing r=mythmon

Comment 4

[Tif Tran]

Attachment: request.txt

Comment 5

[Chris H-C :chutten]

Comment on attachment 9195969 [details]
request.txt

Code is not documentation, so I'm afraid TelemetryEnvironment.jsm#231 is insufficient to meet the requirements for Q6. The Environment is documented here: https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/environment.html

Also, is it necessary to include this pref in userPrefs? Data collected that way does not have an expiry or ownership mechanisms so it's really not a great place to put things unless you absolutely need the information to be present forever in each and every ping that contains an Environment. If this is indeed desired, and you intend to remove this collection after 6 months (or renew it), please write a test that fails if that collection is still present and the Firefox version has advanced to 91.

The alternative would be to collect the value of these prefs in Scalars when they change. Scalars provide ownership and expiry mechanisms and are included in the "main" ping by default, and in custom pings via the record_into_store property.

Comment 6

[Michael Cooper [:mythmon]]

The goal here is to search for very subtle potential issues in the Normandy mechanisms, so using the simplest possible collection is important for us. To that end, adding any extra code to collect this data is not desirable. We don't need it to be present forever or in every ping, but adding a pref listener to record data into a scalar does not sound like the right approach for this. If there is another equally simple (ie, no new code) way to collect the value of these preferences, I'm open to that as well.

We can add a test to fail if this is still present in six months, and I'll file a bug to track that removal. Along with updating the documentation at https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/environment.html#userprefs, will that be enough?

Comment 7

[Chris H-C :chutten]

(In reply to Michael Cooper [:mythmon] from comment #6)

The goal here is to search for very subtle potential issues in the Normandy mechanisms, so using the simplest possible collection is important for us. To that end, adding any extra code to collect this data is not desirable. We don't need it to be present forever or in every ping, but adding a pref listener to record data into a scalar does not sound like the right approach for this. If there is another equally simple (ie, no new code) way to collect the value of these preferences, I'm open to that as well.

We can add a test to fail if this is still present in six months, and I'll file a bug to track that removal. Along with updating the documentation at https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/data/environment.html#userprefs, will that be enough?

Yes, that will do. Though I might grumble a bit under my breath that a collection not needed in every ping will nevertheless present in every ping : ) (it's only for a short while, I understand)

Comment 8

[Tif Tran]

Attachment: request.txt

Comment 9

[Chris H-C :chutten]

Comment on attachment 9196444 [details]
request.txt

DATA COLLECTION REVIEW RESPONSE:

Is there or will there be documentation that describes the schema for the ultimate data set available publicly, complete and accurate?

Yes.

Is there a control mechanism that allows the user to turn the data collection on and off?

Yes. This collection is Telemetry so can be controlled through Firefox's Preferences.

If the request is for permanent data collection, is there someone who will monitor the data over time?

No. This collection will expire within 6 months.

Using the category system of data types on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Category 1, Technical.

Is the data collection request for default-on or default-off?

Default on for pre-release channels only.

Does the instrumentation include the addition of any new identifiers?

No.

Is the data collection covered by the existing Firefox privacy notice?

Yes.

Does there need to be a check-in in the future to determine whether to renew the data?

Yes. :ttran is responsible for renewing or removing the collection before it expires in six months.

---

Result: datareview+

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8b0372660ca7
Added Normandy Do Nothing Prefs for testing r=mythmon

Comment 11

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/8b0372660ca7