986530 - Deploy a test experiment to the nightly population

Status: VERIFIED FIXED

(Firefox Health Report Graveyard :: Client: Desktop, defect)

Description

[Benjamin Smedberg]

We want to deploy a test experiment to nightly users primarily to make sure that the experiment system is working properly. It will have the following characteristics:

* The addon will be http://people.mozilla.org/~bwinton/tile-switcher/
* This changes the frecency ordering of tiles in the about:newtab page, which is something we've already measured using telemetry.
* We want to deploy with a 25% sample to make sure that sampling works reasonably
* Also want to ship with a jsfilter to make sure the filtering and logging works: I suggest a filter that checks the FHR payload and rejects the install if either of the following addons is active:
** tree style tabs "[e-mail]"
** classic theme restorer "ClassicThemeRestorer@ArisT2Noia4dev"

Note: I'm not excluding those because they are bad, just because some nightly users use them and so they make a reasonable test.

Not sure whether to block this on bug 985084; it would be nice to get a better test of that code in the field.

Comment 1

[Benjamin Smedberg]

Heh, this happened.

Comment 2

[Kamil Jozwiak [:kjozwiak]]

Went through verification with the latest Nightly build from the following location:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-04-29-03-02-01-mozilla-central/

- Downloaded the latest Nightly build and ensured "experiments.enabled;true" by default
- Changed "experiments.force-sample-value;0.25" and ensured that the experiment was installed without any issues
- Ensured that "Active:true" under "Experimental Features" in about:support (ensured name, id, description, active and end date were listed)
- Ensured that the "Experiments" container appeared under about:addons and worked without any issues
- Ensured that the "Experiments" container included the latest nightly experiment
- Ensured that the experiment wasn't added into the extensions.bootstrappedAddons quick access cache
- Went through the above test cases using Windows, OSX and Ubuntu