1520252 - "[taskcluster:error] exit status 1" should not be flagged as its own error, nor should it be used for known-intermittent bug-matching suggestions

Status: NEW

(Release Engineering :: Applications: MozharnessCore, defect)

Description

[Daniel Holbert [:dholbert]]

Looking at the OS X and Windows reftest failures on this Try run...
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b2546ed63a5deec9222370611e55cf61281f492

...every single one ends with:

[taskcluster:error] exit status 1

...which TreeHerder highlights alongside the actual error and treats as its own independent failure, as if it were worthy of bug-filing and bug-matching. It suggests three intermittent "[taskcluster:error] exit status $SOME_NUMBER" bug-matches as well.

This all makes the TreeHerder UI more footgunny & harder to interpret. (E.g. it makes it look like we've got known-intermittent bug matches for https://treeherder.mozilla.org/#/jobs?repo=try&revision=9b2546ed63a5deec9222370611e55cf61281f492&selectedJob=221258561 even though we don't.)

EXPECTED RESULTS:
TreeHerder shouldn't highlight this line as a test failure, because it's not. I think it's simply a line of logging for taskcluster recognizing that there was a test failure earlier in the log. This recognition-logging isn't worthy of an independent bug report / bug match in and of itself.

Comment 1

[Daniel Holbert [:dholbert]]

Attachment: screenshot showing the confusing UI

Comment 2

[Phil Ringnalda (:philor)]

Those were good times (for everybody except the people who had to deal with the treeherder/tbpl regex hell), when we could just get treeherder to deal with everyone else's failure to properly log things, weren't they?

The string "[taskcluster:error]" exists entirely for taskcluster to tell treeherder that something is an error and should be highlighted.

What you have there is the reftest harness logging REFTEST TEST-UNEXPECTED-FAIL, mozharness reading the log, deciding based on seeing those that it should exit with a non-zero code, mozharness logging a non-error "WARNING - returning nonzero exit status 1", and taskcluster knowing nothing other than "1" and saying that the run had some sort of failure.

Taskcluster doesn't want to be in the business of parsing logs, and I suspect that we don't want to entirely depend on mozharness always being able to log something that can be parsed anyway, so I think what you probably wanted to file was a mozharness bug along the lines of "Exit with a code other than 1 when something has been logged which will explain to treeherder what the failure was, and teach taskcluster that when it sees that > 1 error code, it should call the overall status of the run a failure but shouldn't [taskcluster:error] about it"