1436701 - [Normandy Telemetry] Pref-experiments - (user-preference-changed-sideload "didResetValue"): "true" when it should be "false"

Status: VERIFIED FIXED

(Firefox :: Normandy Client, defect, P1)

Description

[Adrian Florinescu [:aflorinescu]]

[Environments:]
OsX 10.12 
Win10 x64
Ubuntu 16.04 x64 
Nightly 60.0a1 20180207100355

[Prerequisites:]
You need access to New Admin interface (https://normandy-admin.stage.mozaws.net/control-new)
1. Obtain a copy of Firefox with the SHIELD recipe client system add-on installed. You can check about:support to ensure that you have it.
2. Set the extensions.shield-recipe-client.dev_mode preference to true to run recipes immediately on startup.
3. Set the extensions.shield-recipe-client.logging.level preference to 0 to enable more logging.
4. Set the security.content.signature.root_hash preference to DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90.
5. Set the preference value for extensions.shield-recipe-client.api_url set to https://normandy.stage.mozaws.net/api/v1.

[Steps:]
1. Open Firefox, about:config and select a preference you will want to change using shield preference experiment.
2. Open Control Center: https://normandy-admin.stage.mozaws.net/
3. Create, approve and publish a new recipe of type preference experiments using the first step preference.
4. Open FF with the Shield addon installed and having the pre-requisites set and open the browser console.
5. Open about:config and find the changed preference by the Normandy Preference Experiment.
6. Change the value of the preference to something different than the default value or of the value set by the experiment.
6'. Close Firefox and from prefs.js change the value of the preference to something different than the default value or of the value set by the experiment.
7. Restart FF.
8. Open about:telemetry
9. In the left panel, in the current ping data, locate Events. (if there are no events in the current ping data, then Left top of the page, click on main, select for "ping data source": "Archived ping data" and find the main that contains the events).
10. In the Events data, in the top right corner select dynamic. (default is parent)
11. Verify in the telemetry event that the values for unenroll event.


[Actual Result:]
6. 
26265 normandy unenroll preference_study qa1 {"didResetValue": "true", "reason": "user-preference-changed"}
6'.
2703 normandy unenroll preference_study qa1 {"didResetValue": "true", "reason": "user-preference-changed-sideload"}

[Expected Result:]
The value for "didResetValue" should be false, since Normandy did not reset the value in either of these cases, it was user modified.


[Additional Notes:]
There is another scenario for the "not-seen" reason, when the preference experiment expires due to recipe not being available anymore and in the same session the user modifies or has the value modified, in which case the value for "didResetValue" will also be true, when it is actually false.
They are slightly different scenarios, not sure if this should be a separate bug, since AFAIK, this issue revolves around the fact that the actual reset only happens at start-up. Since this is an edge-case, I did not find it relevant to have a separate bug for it.

Comment 1

[Michael Cooper [:mythmon]]

Attachment: Bug 1436701 - Handle resetValue / didResetValue correctly in Normany preference experiments

Review commit: https://reviewboard.mozilla.org/r/218846/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/218846/

Comment 2

[:Gijs (he/him)]

Comment on attachment 8949507 [details]
Bug 1436701 - Handle resetValue / didResetValue correctly in Normany preference experiments

https://reviewboard.mozilla.org/r/218846/#review224780

Thanks!

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/79a099532949
Handle resetValue / didResetValue correctly in Normany preference experiments r=Gijs

Comment 4

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/79a099532949

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/79a099532949

Comment 6

[Adrian Florinescu [:aflorinescu]]

For: "user-preference-changed-sideload" the fix is ok:
1361 	normandy 	unenroll 	preference_study 	telemetry-exp 	{"didResetValue": "false", "reason": "user-preference-changed-sideload"} 

For: "user-preference-changed", I believe the didResetValue should also be false:
15869 	normandy 	unenroll 	preference_study 	telemetry-exp 	{"didResetValue": "true", "reason": "user-preference-changed"}

Comment 7

[Michael Cooper [:mythmon]]

Reviewboard doesn't like me having two patches on one bug. I'm closing this bug and filed bug 1437668 as a follow up for the user-preference-changed version of the problem. Sorry about that!

Comment 8

[Adrian Florinescu [:aflorinescu]]

ok, lets then change the scope of this bug as a placeholder for the user-preference-changed-sideload, which the patch actually fixes. I'll double check it again when bug 1437668 lands.

Comment 9

[Adrian Florinescu [:aflorinescu]]

re-verified the fix on 60.0a1 2018-02-14 on Ubuntu 16.04 and OsX 10.12, with bug 1437668 landed:
2890 	normandy 	unenroll 	preference_study 	telemetry-exp 	{"didResetValue": "false", "reason": "user-preference-changed-sideload"}