1594422 - [meta] CFR Machine Learning Experiment

Status: RESOLVED WONTFIX

(Firefox :: Messaging System, enhancement)

Comment 1

[Kate Hudson :k88hudson]

Attachment: Technical documentation

Comment 2

[fwollsen]

Attachment: [e-mail]

Uploading control add-on v0.1.0 for test-signing

Comment 3

[fwollsen]

Attachment: [e-mail]

Uploading treatment add-on v0.1.0 for test-signing

Comment 4

[fwollsen]

Attachment: [e-mail]

Uploading control add-on v0.1.1 for test-signing

Comment 5

[fwollsen]

Attachment: [e-mail]

Uploading treatment add-on v0.1.1 for test-signing

Comment 6

[mlopatka]

@jvehent These are the control and treatment condition addons for the CFR study. Requesting signing whenever/however possible so that we can have preliminary QA performed on just the client-side functionality in parallel with continued refinement of the end-to-end testing plan and further signoffs as per the RRA.

Comment 7

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 8

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 9

[fwollsen]

Attachment: [e-mail]

Comment 10

[fwollsen]

Attachment: [e-mail]

Comment 11

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 12

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 13

[fwollsen]

Attachment: [e-mail]

Comment 14

[fwollsen]

Attachment: [e-mail]

Comment 15

[Michael Cooper [:mythmon]]

By the version number bump I assume that this is intended for a final signature now? Have QA and a Firefox Peer signed off on this version?

Comment 16

[fwollsen]

By the version number bump I assume that this is intended for a final signature now? Have QA and a Firefox Peer signed off on this version?

It is intended for final QA, but not production signing yet. For us, v1.0.0 means fully implemented. Subsequent rounds of QA will lead to 1.0.1, 1.0.2 etc until a version has been green-lighted by QA, after which no final version bump will be made, only a re-signing with the production key. This way Normandy can ship the same build that was green-lighted by QA, removing any risks associated with rebuilding the add-on.

Comment 17

[Michael Cooper [:mythmon]]

Sounds great. Thanks for the info.

Comment 18

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 19

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 20

[Ilana]

Attachment: Messaging System Personalization Experiment 1 - data collection review form.txt

Needed ASAP

Comment 21

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Is https://github.com/mozilla/messaging-system-personalization-experiment-1-addon/blob/master/docs/TELEMETRY.md the correct public documentation? Please update either the documentation or the request to match what you plan to collect; thanks!

Comment 22

[Ilana]

(In reply to Tim Smith 👨‍🔬 [:tdsmith] from comment #21)

Is https://github.com/mozilla/messaging-system-personalization-experiment-1-addon/blob/master/docs/TELEMETRY.md the correct public documentation? Please update either the documentation or the request to match what you plan to collect; thanks!

Yes. Everything not listed in the data steward review is a telemetry feature that is already collected in release or a computed value based on that data - does that need to be added to the doc?

Comment 23

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Comment on attachment 9118915 [details]
Messaging System Personalization Experiment 1 - data collection review form.txt

In general, yes, a duplicate collection through a different mechanism is also reviewable.

But let's just accept https://github.com/mozilla/messaging-system-personalization-experiment-1-addon/blob/3ec5d3f11d6674c86886dcfba287ccb2877f00fa/docs/TELEMETRY.md as the response to question 5!

data-review+
--
1) Is there or will there be **documentation** that describes the schema for the ultimate data set in a public, complete, and accurate way?

Yes, at https://github.com/mozilla/messaging-system-personalization-experiment-1-addon/blob/3ec5d3f11d6674c86886dcfba287ccb2877f00fa/docs/TELEMETRY.md. The doctype is `messaging-system-personalization-experiment-1-update` and the pings include clientIds but not environments.

2) Is there a control mechanism that allows the user to turn the data collection on and off?

Yes, the Firefox telemetry opt-out or Normandy opt-out.

3) If the request is for permanent data collection, is there someone who will monitor the data over time?

n/a

4) Using the **[category system of data types](https://wiki.mozilla.org/Firefox/Data_Collection)** on the Mozilla wiki, what collection type of data do the requested measurements fall under?

Categories 1 and 2.

5) Is the data collection request for default-on or default-off?

Default-on.

6) Does the instrumentation include the addition of **any *new* identifiers**?

No.

7) Is the data collection covered by the existing Firefox privacy notice?

Yes.

8) Does there need to be a check-in in the future to determine whether to renew the data?

No; the telemetry will be collected for the lifetime of the study.

9) Does the data collection use a third-party collection tool?

No.

Comment 24

[Michael Feldman (use NI)]

No additional legal review needed. Alicia Gray will sign off for both Trust/Security and legal.

Comment 25

[Victor Ng [:vng] [:crankycoder]]

Attachment: [e-mail]

Comment 26

[Victor Ng [:vng] [:crankycoder]]

Attachment: [e-mail]

Comment 27

[Victor Ng [:vng] [:crankycoder]]

Uploading control and treatment add-on v1.0.1. This changes the name of the cfr-ml-model collection to cfr-ml-models

Comment 28

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 29

[Michael Cooper [:mythmon]]

Attachment: [e-mail]

Comment 30

[Patriciu Pop [:ppop], Ecosystem QA]

[PI-438] CFR personalization experiment
Firefox Release 72.x

We have finished testing the CFR personalization experiment.

We have found the following issue which we consider a blocker for the experiment:
Bug 1608803 The user is wrongly unenrolled from the study due to unexpected “CAUGHT ERROR” exception after browser restart - https://bugzilla.mozilla.org/show_bug.cgi?id=1608803

QA’s recommendation: RED - DON’T SHIP

Reasoning:

• The found issue is considered blocker for the release from QA perspective due to the fact that users enrolled in the study will be unenrolled after a browser restart.
• Testing was blocked for the treatment branch (main branch) of the add-on therefore we were not able to test any of the functionalities specific to this branch.
• The "treatment" branch scores are not updated after each “cfr-ml-model” Remote Settings bucket update, due to the same "Invalid telemetry payload" error displayed in the console. Might have the same root cause as the issue logged, but we cannot say for sure.
• We haven’t received steps to naturally trigger the CFR recommendation and all the testing performed until now was based on manually overriding the prefs values.

What needs to be done?

• In order to receive a GREEN sign-off we will need the blocker issue to be fixed and we will require testing steps to update the value of the “browser.messaging-system.personalized-cfr.scores” preference in order to naturally trigger a CFR recommendation.

Testing Summary:

• Verified that the control add-on is installed and it correctly modifies the targeted preferences.
• Tried to install the treatment add-on via Normandy and encountered Bug 1608803 (https://bugzilla.mozilla.org/show_bug.cgi?id=1608803).
• We could not test the add-on using Firefox Dev Edition due to Bug 1608832 (https://bugzilla.mozilla.org/show_bug.cgi?id=1608832) that blocks Normandy stage recipes from executing on the channel.
• Tried to change the value of the “browser.messaging-system.personalized-cfr.scores” preference by force updating the “cfr-ml-model” collection, adding an override to shorten the polling period, and waiting an hour for the collection to naturally update without success.

Tested Platforms:

• Windows 10 x64
• macOS 10.15.1
• Ubuntu 18.04

Tested Firefox versions:

• Firefox Nightly 74.0a1
• Firefox Unbranded Release 72.0.2
• Firefox Dev Edition 73.0b4

Regards,
Patriciu