1170418 - Jetpack test runs show TypeErrors in the log, but the run is green

Status: RESOLVED INCOMPLETE

(Add-on SDK Graveyard :: General, defect)

Description

[Erik Vold [:erikvold] (please needinfo? me)]

The treeherder UI is not catching bug 1170400, for some reason the bug is ignored and the tests are shown as green some of the time, like in https://treeherder.mozilla.org/logviewer.html#?job_id=3290950&repo=fx-team

Comment 1

[Phil Ringnalda (:philor)]

That's actually "Treeherder thinks that something is an error which the jetpack test harness does not think is an error." Buildbot is doing the right thing, because years ago in bug 679490 jetpack told it "if the exit code is 0, the run passed." Treeherder is doing the right thing by saying that log is a green run, because if you look at the raw log, the fourth line is "results: success (0)" which says that the run was green. Treeherder is also doing the right thing by saying that line might be an error, since in some contexts TypeError is an error. So you have one of two things: "I want to make the jetpack test harness declare that when that happens, the test failed," a jetpack bug, or, "that happening is otherwise fine, but it spams the log and confuses people into thinking that it is a failure when in fact it is fine, so I want to catch it in utils.js and just return null from getOwnerWindow() when it's called with a null tab," a jetpack bug.

Comment 2

[Ed Morley [:emorley]]

(fwiw if you file bugs against Treeherder, they need to be in a Treeherder component not a random releng component :-) not that this is a treeherder bug, for the reasons in comment 1 - but for future bugs...)

Comment 3

[Firefox Bug Husbandry Bot]

https://bugzilla.mozilla.org/show_bug.cgi?id=1399562