Closed Bug 1502078 Opened 8 years ago Closed 8 years ago

Expired preference experiment do not set the initial state in case of non-existent preferences

Categories

(Firefox :: Normandy Client, defect)

defect
Not set
major

Tracking

()

RESOLVED DUPLICATE of bug 1502410
Tracking Status
firefox63 --- fixed
firefox64 --- fixed
firefox65 --- fixed

People

(Reporter: danibodea, Unassigned)

References

Details

(Keywords: regression)

[Description]: - Expired preference experiment do not set the initial state in case of non-existent preferences. [Affected versions]: -(not affected) Nightly v62.0a1 (2018-05-31) build ID: 20180531220139 -(affected) Nightly v65.0a1 (2018-10-25) -(affected) Release v63.0 [Affected platforms]: - most likely all, tested on Windows 10 [Pre-conditions:] user_pref("security.content.signature.root_hash", "DB:74:CE:58:E4:F9:D0:9E:E0:42:36:BE:6C:C5:C4:F6:6A:E7:74:7D:C0:21:42:7A:03:BC:2F:57:0C:8B:9B:90"); user_pref("app.normandy.api_url", "https://normandy.stage.mozaws.net/api/v1"); user_pref("app.normandy.dev_mode", true); user_pref("app.normandy.logging.level", 0); [Steps to reproduce]: 1. Create/Publish a pref-experiment that sets a preference that doesn’t exist. 2. Run a firefox client with the pre-conditions. 3. Disable the pref-experiment 4. Restart the client. [Expected result]: 2. Preference is created and the experiment value is set. 4. Experiment expires -> not seen, Preference and its value are removed. [Actual result]: - the pref. Experiment is shown as not-seen / expired. - the preference still exists. - the preference value is still the one set by the experiment. [Additional Information:] for the non-existent preference experiment: -preference not removed, value not reset -normandy start-up preference not removed: app.normandy.startupExperimentPrefs.SVpref -Value for shield-preference-experiments.json: {"experidani":{"name":"experidani","branch":"default","expired":true,"lastSeen":"2018-10-25T12:19:38.013Z","preferenceName":"SVpref","preferenceValue":100,"preferenceType":"integer","previousPreferenceValue":null,"preferenceBranchType":"default","experimentType":"exp"}, for the preference experiment that affects existent preferences: -preference value reset to initial value -normandy start-up preference not removed: app.normandy.startupExperimentPrefs.accessibility.delay_plugin_time -Value for shield-preference-experiments.json: "experidani2":{name":"experidani2","branch":"var1","expired":true,"lastSeen":"2018-10-25T12:35:44.086Z","preferenceName":"accessibility.delay_plugin_time","preferenceValue":500,"preferenceType":"integer","previousPreferenceValue":10000,"preferenceBranchType":"default","experimentType":"exp"}} [Regression range]: Not available.
The revision in question is this one: https://normandy.stage.mozaws.net/api/v3/recipe_revision/973/ The arguments causing this are { "preferenceBranchType": "default", "slug": "experidani", "experimentDocumentUrl": "https://www.google.com/", "isHighVolume": false, "preferenceName": "SVpref", "preferenceType": "integer", "isEnrollmentPaused": false, "branches": [ { "slug": "default", "value": 100, "ratio": 1 }, { "slug": "variant1", "value": 11, "ratio": 1 }, { "slug": "variant2", "value": 22, "ratio": 1 }, { "slug": "variant3", "value": 33, "ratio": 1 } ] }
I was able to reproduce this following the steps in comment 0. While doing so I noticed the traceback for bug 1502182, so possible these two bugs are related. I tried to write a test for this, but haven't been able to reproduce in a mochitest. I'm going to switch gears to 1502182, and see if I can get reproduce that in tests and fix it. Perhaps it will also fix this bug.
See Also: → 1502410
Investigating the scenario for the existent preferences for which a pref-experiment is conducted, bug 1502410 was logged. The same regression range for bug 1502410 is confirmed for this bug as well: last good build: 63.0a1 (2018-07-14) first bad build: 63.0a1 (2018-07-15) https://hg.mozilla.org/mozilla-central/pushloghtml?startdate=105+days+ago&enddate=103+days+ago
Tracked as a blocking issue for 63 until we fully understand the impact.
Adrian, Michael, does it affect both Desktop and Fennec or only Desktop?
Flags: needinfo?(mcooper)
Flags: needinfo?(adrian.florinescu)
Normandy is not available for mobile at the current time, therefore it just affects Desktop.
Flags: needinfo?(mcooper)
Flags: needinfo?(adrian.florinescu)
I'm now convinced that this bug and bug 1502410 are duplicates of eachother. There is more progress made in the other bug, so I'm duping this one to it. I'll transfer over the tracking metadata as well.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → DUPLICATE
These are my steps on the try build from bug 1502410: 1. Create and Publish a pref-experiment that sets a preference that doesn’t exist. 2. Run a firefox client with the pre-conditions. 3. Disable the pref-experiment 4. Restart the client. [Actual result]: - the pref-experiment is shown as not-seen / expired. - the preference still exists. - the preference value is still the one set by the experiment. 5. Restart the client again. - the pref. is ONLY NOW removed
Furthermore, I have tested the situation where the recipe is disabled, the bug is reproduced on the affected build, then the fixed build is opened to see if the issue is fixed for the users that reproduced it with older recipes. STR: 1. Create/Publish a pref-experiment that sets a preference that doesn’t exist. 2. Run the affected build (latest nightly v64.0a1 from 2018-10-30) with the pre-conditions. Notice that: Preference is created and the experiment value is set. 3. Disable the pref-experiment 4. Restart the affected build 2 times to be sure that the issue reproduced. Notice that: The current issue reproduces (the created pref is not removed). 5. Take the try build from bug 1502410 and open it using the same profile. Notice that the pref-experiment pref is not yet removed. 6. Restart the client (fixed try build). Notice that: Only now the pref-experiment created pref gets removed. As a conclusion, having the fix applied for the users that were enrolled in pref-experiments that were disabled and the issue occurred, the issue is fixed after 2 restarts. (The incorrectly remaining pref is removed.)
You need to log in before you can comment on or make changes to this bug.