We suspect Normandy Pref Experiments aren't correctly setting or reporting that they've set prefs. In bug 1499552 I added a default pref with value false for us to set via a Pref Experiments, but I think we should also have a pref with default true to set via the dummy Pref Experiment. We're seeing about a 25% failure rate in the Pref Experiment, so maybe one of these four cases isn't working as expected? So we should add another default pref with a default value of true so we can set that via a dummy Pref Experiment.
The Webrender Pref Experiment is reporting its pref via telemetry and that is reporting a different value than what the Normandy experiments telemetry indicates should be being seen. In Bug 1499552 I added a pref and telemetry reporting in preparation for running a dummy Pref Experiment to verify that prefs set are reported as set, but I realised I missed a case; where there's an existing default pref set, we're only covering the case where that default pref has value false. We're seeing a ~25% failure rate in the pref reporting. So we should cover the other three cases in the dummy Pref Experiment, just in case it's one of these four cases that is failing. So here I add another dummy pref with a default value of true, and I report that via telemetry. I rename the pref I added yesterday to make things consistent.
I realised the data collection I requested yesterday in bug 1499552 missed a case, so requesting a second pref to be collected.
Attachment #9018406 - Flags: review?(francois)
Comment on attachment 9018406 [details] Data collection request The changes in this bug do not change any of the answers in the original data review: https://bugzilla.mozilla.org/show_bug.cgi?id=1499552#c3
Attachment #9018406 - Flags: review?(francois) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/38c1fc84fcaf Add dummy pref with default value true, report via telemetry. r=mythmon
Since our study has concluded, I backed out this telemetry in bug 1502779.
You need to log in before you can comment on or make changes to this bug.