Closed Bug 1021002 Opened 6 years ago Closed 6 years ago

Nightly test experiment logging doesn't show failures due to incompatible extensions: because the jsfilter isn't working!

Categories

(Firefox Health Report Graveyard :: Client: Desktop, defect)

defect
Not set

Tracking

(firefox31- wontfix, firefox32+ verified, firefox33 verified)

VERIFIED FIXED
Firefox 33
Tracking Status
firefox31 - wontfix
firefox32 + verified
firefox33 --- verified

People

(Reporter: benjamin, Assigned: gfritzsche)

References

Details

(Whiteboard: p=5 s=33.1 [qa!])

Attachments

(1 file)

Looking through the dashboard for the nightly deployment of the tile-switcher experiment, I expected to see activations that were rejected because the user had tree-style-tabs or classic-theme-restorer, but I don't see any indication of that.

Visit http://bsmedberg.github.io/telemetry-experiments-dashboard/ and add "first date: 2014-04-25" and nightly release channel

Failed Activations

Reason	#
REJECTED,endTime	3882459
REJECTED,sample	3224130
INSTALL_FAILURE	12069
REJECTED,failedStart	3559
REJECTED,startTime	1316
REJECTED,Error: Service failed to initialize.	250
REJECTED,Error: Error(s) encountered during statement execution.	101
REJECTED,maxActiveSeconds	47
REJECTED,TypeError: this._providerManager is null	34
REJECTED,Error: Not initialized.	11
REJECTED,channel	11
REJECTED,TypeError: reporter is null	10
REJECTED,TypeError: this.log is not a function	1
REJECTED,TypeError: this._storage is null	1

I would expect to see some entries like

REJECTED,jsfilter-false
REJECTED,jsfilter-threw,Error: Tree-style tabs present

Kamil, do you have time to try this and see what actually shows up in the telemetry log?

STR:
* Clone and adjust the experiment manifest to get rid of the sample and change the dates to the present
* Run the browser and force check for experiment updates. In the browser console:

Cu.import("resource:///modules/experiments/Experiments.jsm");
Experiments.instance().updateManifest();

* After the update is finished, get the telemetry log entries by running this in the browser console:

Cu.import("resource://gre/modules/TelemetryLog.jsm");
TelemetryLog.entries();
Flags: needinfo?(kamiljoz)
Flags: firefox-backlog+
- Installed the "tree-style-tabs" add-on (same results using the classic-theme-restorer add-on)
- Followed the instructions from comment #0 (removing the sample rate, etc..)
- Once the experiment was installed, typed in "TelemetryLog.entries().toSource();" into the Browser Console. Received the following log entries:

- ["EXPERIMENT_ACTIVATION", 52487, "ACTIVATED", "tile-switcher@experiments.mozilla.org"]
- ["EXPERIMENT_TERMINATION", 389463, "ADDON_UNINSTALLED", "tile-switcher@experiments.mozilla.org"]
- ["EXPERIMENT_ACTIVATION", 400009, "ACTIVATED", "tile-switcher@experiments.mozilla.org #2"]
- ["EXPERIMENT_TERMINATION", 692040, "ADDON_UNINSTALLED", "tile-switcher@experiments.mozilla.org #2"]
- ["EXPERIMENT_ACTIVATION", 731416, "ACTIVATED", "tile-switcher@experiments.mozilla.org #3"]

I never received an error message indicating that "tree-style-tabs" or "classic-theme-restorer" have been installed. I tried it a few times with the same results. (Installing/Removing/Installing)

Benjamin, let me know if these are the correct log entries that you were looking for!
Flags: needinfo?(kamiljoz)
Crap, that means that the jsfilter isn't working for some reason. Thanks Kamil, this is an important bug. This will also affect 31, but it may not matter because we don't have any planned experiments for 31.
Summary: Nightly text experiment logging doesn't show failures due to incompatible extensions → Nightly text experiment logging doesn't show failures due to incompatible extensions: because the jsfilter isn't working!
I'll keep my eye on this bug and test it as soon as it's fixed. I'll also add this case into "Important things to check" under the wiki and will personally start testing this case every time I go through experiments testing!
Assignee: nobody → georg.fritzsche
Status: NEW → ASSIGNED
Whiteboard: p=5 [qa+] → p=5 s=33.1 [qa+]
Summary: Nightly text experiment logging doesn't show failures due to incompatible extensions: because the jsfilter isn't working! → Nightly test experiment logging doesn't show failures due to incompatible extensions: because the jsfilter isn't working!
healthreporter.collectAndObtainJSONPayload() returns a stringified payload, which meant we were double-stringifying that payload.

Of course this didn't show up in the tests because we mocked it to return an object payload instead...

I threw in two minor bonus fixups (unused code + task return handling).
Attachment #8439479 - Flags: review?(benjamin)
Attachment #8439479 - Flags: review?(benjamin) → review+
https://hg.mozilla.org/mozilla-central/rev/a51e772eb23d
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 33
Assigning this to Kamil based on comment 3.
QA Contact: kamiljoz
Used the following build for verification:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-17-03-02-03-mozilla-central/

Logs received using "experiments.logging.level;0" once "Experiments.instance().updateManifest();" is run:

* 1403015002143	Browser.Experiments.Experiments	TRACE	Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","tile-switcher@experiments.mozilla.org","jsfilter-threw","Tree-style tabs present"]
* 1403015279358	Browser.Experiments.Experiments	TRACE	Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","tile-switcher@experiments.mozilla.org","jsfilter-threw","Tree-style tabs present"]
* 1403016034200	Browser.Experiments.Experiments	TRACE	Experiments #0::evaluateExperiments() - added EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","tile-switcher@experiments.mozilla.org","jsfilter-false"]

Logs received using "TelemetryLog.entries().toSource();"

* ["EXPERIMENT_ACTIVATION", 67515, "REJECTED", "tile-switcher@experiments.mozilla.org", "jsfilter-threw", "Tree-style tabs present"]
* ["EXPERIMENT_ACTIVATION", 344734, "REJECTED", "tile-switcher@experiments.mozilla.org", "jsfilter-threw", "Tree-style tabs present"]
* ["EXPERIMENT_ACTIVATION", 74578, "REJECTED", "tile-switcher@experiments.mozilla.org", "jsfilter-false"]
* ["EXPERIMENT_ACTIVATION", 251437, "REJECTED", "tile-switcher@experiments.mozilla.org", "jsfilter-false"]]"

I also ensured that once "tree-style-tabs" or "classic-theme-restorer" are removed, the current experiment is installed and logged correctly.

The one thing that I noticed is that the logs don't indicate why the experiment was terminated while using "classic-theme-restorer". Not sure if this is an issue but it would be useful to have that information logged the same way it's being done for "tree-style-tabs".

Benjamin, should we fix the above to indicate that it was terminated because of "classic-theme-restorer present"??
Flags: needinfo?(benjamin)
That was an intentional choice when implementing this first filter function so that we could test both the "throw" and "return false" codepaths. This is working as expected. Future experiements will probably just use the throw codepath for the extra logging detail.
Status: RESOLVED → VERIFIED
Flags: needinfo?(benjamin)
Comment on attachment 8439479 [details] [diff] [review]
Fix FHR JSON payload handling

[Approval Request Comment]
Bug caused by (feature/regressing bug #): Telemetry experiments.
User impact if declined: jsfilters broken for telemetry experiments.
Testing completed (on m-c, etc.): Verified on Nightly.
Risk to taking this patch (and alternatives if risky): Low-risk - well-understood once the underlying issue was located. 
String or IDL/UUID changes made by this patch: None.
Attachment #8439479 - Flags: approval-mozilla-aurora?
Not tracking since it won't be a problem with 31.
Any update on the approval here?
Flags: needinfo?(release-mgmt)
Comment on attachment 8439479 [details] [diff] [review]
Fix FHR JSON payload handling

Aurora approval granted.
Attachment #8439479 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
Flags: needinfo?(release-mgmt)
Blocks: 983226
Went through verification for fx32 using the following build:
- http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-06-25-00-40-01-mozilla-aurora/

I went through the same test cases outlined in comment # 8 without any issues. Ensured when either "tree-style-tabs" or "classic-theme-restorer" are installed, the experiment is NOT installed and is correctly logged. Extracted the logs using (from comment #0):

Cu.import("resource://gre/modules/TelemetryLog.jsm");
TelemetryLog.entries().toSource();

The correct errors also appeared in the browser console when installing the experiment. I also ensured that the following is being logged:

- ["EXPERIMENT_TERMINATION", 31011, "EXPIRED", "tile-switcher@experiments.mozilla.org #3"]
- ["EXPERIMENT_ACTIVATION to TelemetryLog: ["REJECTED","tile-switcher@experiments.mozilla.org #4","jsfilter-threw","Tree-style tabs present"]
- ["EXPERIMENT_ACTIVATION", 31640, "REJECTED", "tile-switcher@experiments.mozilla.org #6", "jsfilter-false"] (classic-theme-restorer)
- ["EXPERIMENT_ACTIVATION", 107781, "ACTIVATED", "tile-switcher@experiments.mozilla.org #5"]]"
- ["EXPERIMENT_TERMINATION", 5922, "ADDON_UNINSTALLED", "tile-switcher@experiments.mozilla.org #4"]
Whiteboard: p=5 s=33.1 [qa+] → p=5 s=33.1 [qa!]
Product: Firefox Health Report → Firefox Health Report Graveyard
You need to log in before you can comment on or make changes to this bug.