1518417 - Users are stuck in experiments when Normandy is disabled

Status: NEW

(Firefox :: Normandy Client, defect)

Description

[Daniel Bodea [:danibodea]]

[Note]:

• The experiment pref values do not revert to the original state when disabling Telemetry or Normandy all together

[Affected versions]:

• Release v64.0
• Beta v65.0b7
• Nightly v66.0a1

[Affected platforms]:

• tester on Windows 7 and 10, most probably all

[Steps to reproduce]:

1. Create a pref experiment recipe
2. Run Firefox with the required prerequisites set.
3. Notice that the experiment pref it's changed value according to the recipe.
   4.1 Disable Telemetry by setting the "datareporting.healthreport.uploadEnabled" pref as false from the "about:config" page.
   OR
   4.2 Untick the "Allow Nightly to send technical and interaction data to Mozilla" option from the "about:preferences#privacy" page.
   OR
   4.3 Disable Normandy by setting the "app.normandy.enabled" pref to false from the "about:config" page.
4. Restart Firefox.

[Expected result]:

• The experiment pref regains its original value.

[Actual result]:

• The users are stuck into the state in which they were at the moment when Normandy was disabled.

[Regression range]:

• This is not a recent regression.

[Additional notes]:

• Most likely studies fall into the same category: they will not unenroll once Normandy is turned off.

Comment 1

[Daniel Bodea [:danibodea]]

As a realistic (actually real) example, the recipe "pref-flip-defaultoncookierestrictions-1506704 / Cohort 3" has had a negative retention. For any user that somehow disabled Normandy, while this pref-flip was on, that user is stuck with a bad functioning firefox.

Comment 2

[Adrian Florinescu [:aflorinescu]]

Raising this to major, since I think this is a valid point to be considered and discussed upon.

Comment 3

[Raluca Popovici, Desktop QA]

This issue is still reproducible on the latest Firefox Nightly 96.0a1 version, updating severity to match current ranking.

Comment 4

[:Gijs (he/him)]

Hey mythmon, if this is S2/P2, is this likely to be prioritized soon? It looks pretty bad to me - I had assumed we unenrolled experiments and maybe rollouts if users disable the feature?

Comment 5

[Michael Cooper [:mythmon]]

I don't agree that this is S2, and the P2 priority is from several years ago, so I don't think it is relevant right now.

Quoting the BMO user guide:

S2 (Serious) Major functionality/product severely impaired and a satisfactory workaround does not exist
S3 (Normal) Blocks non-critical functionality and a work around exists

I don't think that disabling Normandy via about:config is major functionality, nor does it severely impair the browser. There is a workaround: In the case of studies they can still be disabled in about:studies. In the case of rollouts, the rollouts can be removed manually, however we don't support ending rollouts this way.

Given that this is non-critical functionality, and there are work-arounds I think this should be at most S3. Given that the existing prioritization is very stale, I'm going to clear it for the time being.