Closed Bug 1001787 Opened 6 years ago Closed 6 years ago

Telemetry experiments: experiment re-enabling after being removed once FX is restarted

Categories

(Firefox Health Report Graveyard :: Client: Desktop, defect)

defect
Not set

Tracking

(firefox30 unaffected, firefox31+ verified, firefox32 verified)

VERIFIED FIXED
Firefox 32
Tracking Status
firefox30 --- unaffected
firefox31 + verified
firefox32 --- verified

People

(Reporter: kjozwiak, Assigned: gfritzsche)

References

Details

(Whiteboard: p=3 s=it-32c-31a-30b.1 [qa!])

Attachments

(3 files)

Experiments are being re-enabled once firefox has been restarted even though they've been removed by the user.

video of the issue:
- http://youtu.be/zSFgH6QiliE

Steps to reproduce the issue:

* Install the latest Nightly
* Go into "about:config" and add "experiments.force-sample-value = 0.25"
* Once added, enable/disable the "experiments.enabled" preference and the experiment should be installed
* Once installed, go into "about:addons" and remove the experiment by selecting "Remove"
* Once removed, completely close Firefox Nightly
* Re-Open Firefox Nightly and go to "about:support" (You'll notice the experiment is enabled again "Active = true")
* Go into "about:addons" and you'll also notice that the experiment is re-enabled even though it was removed earlier by the user

Current Behaviour:

- When Firefox is restarted, the previously removed experiment is re-enabled even though the user has removed it via the add-ons manager

Expected Behaviour:

- Once the experiment has been removed by the user, it shouldn't be re-enabled when Firefox is started. Only new experiments should be allowed to be installed and not the ones that have been removed.
Blocks: 992258
Assignee: nobody → georg.fritzsche
Flags: firefox-backlog?
Flags: firefox-backlog? → firefox-backlog+
Whiteboard: p=3
Add test coverage for this by:
* uninstalling an experiment addon
* faking a restart
* make sure nothing is active
Attachment #8414388 - Flags: review?(benjamin)
This fixes ExperimentEntry.stop() not propagating changes, which resulted in us not triggering a save of the experiments state.
Attachment #8414392 - Flags: review?(benjamin)
We expect all historic experiments to have a valid endDate. Setting an experiment to disabled and then calling into reconcileAddonState can trigger calls into Experiments.getExperiments() via the AddonManager, so we need to set the endDate before that.

Note: i didn't have luck triggering this specific behavior in the xpcshell-test so far, but verified manually.
Attachment #8414395 - Flags: review?(benjamin)
OS: Mac OS X → All
Hardware: x86 → All
Version: unspecified → Trunk
(In reply to Georg Fritzsche [:gfritzsche] from comment #1)
> Created attachment 8414388 [details] [diff] [review]
> Test addon uninstall & restart

Note that this also adds a common cleanup step to every task in test_api.js, to avoid stale addon state etc.
I forgot to move this to a separate patch.
Attachment #8414388 - Flags: review?(benjamin) → review+
Attachment #8414392 - Flags: review?(benjamin) → review+
Attachment #8414395 - Flags: review?(benjamin) → review+
Status: NEW → ASSIGNED
Whiteboard: p=3 → p=3 s=it-32c-31a-30b.1 [qa?]
Whiteboard: p=3 s=it-32c-31a-30b.1 [qa?] → p=3 s=it-32c-31a-30b.1 [qa+]
QA contact required for resolved Bug 1001787.
Flags: needinfo?(jbecerra)
Flags: needinfo?(jbecerra)
QA Contact: kamiljoz
Went through the verification process using the latest Nightly and Aurora builds from the following location:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-05-01-03-02-02-mozilla-central/
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-05-01-00-40-04-mozilla-aurora/

- Disabled the experiment using the three different "Remove" routes (context menu, preferences or the button next to the experiment)
- Ensured that when the experiment is disabled, "Active" under "about:support" appears as "false"
- Ensured that when you re-start Firefox, the experiment stay's disabled, double checked both "about:addons" and "about:support"
- Ensured that "experiments.activeExperiment" appears as "false" when the experiment has been disabled
- Ensured that "experiments.enabled" is still enabled when an experiment has been removed/disabled
- Enusred that the experiment wasn't added into "extensions.bootstrappedAddons"
- Ensured that you cannot install the experiment that has been disabled (contacting the staging server and trying to get the experiment installed again while it's already disabled)
- Moved the computers date one day at a time and ensured that once the time limit has been reached, the experiment was disabled. Checked "about:config", "about:support" and "about:addons"
- Ensured that the experiment is still disabled after restarting the machine several times

This is still an issue with the latest Aurora. Once Firefox is restarted, the experiment is re-enabled. Will this fix be pushed into Aurora?
Comment on attachment 8414392 [details] [diff] [review]
Fix experiment addon uninstall not triggering a save

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Telemetry experiments
User impact if declined: experiment enabled after restart despite of user disabling it.
Testing completed (on m-c, etc.): verified on m-c, test coverage
Risk to taking this patch (and alternatives if risky): Low, i think the relatively simple fixes here are well enough understood.
String or IDL/UUID changes made by this patch: None.
Attachment #8414392 - Flags: approval-mozilla-aurora?
Comment on attachment 8414395 [details] [diff] [review]
Fix not setting the endDate on historic experiments early enough.

[Approval Request Comment]
See above.
Attachment #8414395 - Flags: approval-mozilla-aurora?
Comment on attachment 8414388 [details] [diff] [review]
Test addon uninstall & restart

[Approval Request Comment]
See above - test coverage.
Attachment #8414388 - Flags: approval-mozilla-aurora?
Duplicate of this bug: 1001529
Duplicate of this bug: 1001529
Attachment #8414392 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Attachment #8414395 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Attachment #8414388 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Went through verification using the following build:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-05-07-00-40-06-mozilla-aurora/

- Went through the test cases in comment #8 without any issues
- Ensured that the tiles are appearing under about:newtab using CTRL + T, File -> New Tab & typing about:newtab inside the URI
Status: RESOLVED → VERIFIED
Whiteboard: p=3 s=it-32c-31a-30b.1 [qa+] → p=3 s=it-32c-31a-30b.1 [qa!]
Product: Firefox Health Report → Firefox Health Report Graveyard
You need to log in before you can comment on or make changes to this bug.