1191378 - Marionette job shows as success but contains failed steps

Status: RESOLVED DUPLICATE of bug 1156070

(Remote Protocol :: Marionette, defect)

Description

[Jonathan French (:jfrench)]

Split off from unrelated TC bug 1190523, the Treeherder and Logviewer view of this Marionette test looks like a normal successful job, except it contains a 'show successful steps' - which contains failed steps.
https://treeherder.allizom.org/logviewer.html#?job_id=9175861&repo=mozilla-inbound

Ed suggested there might be a small adjustment possible on the harness side, to produce a blob that would include a suitable property (eg. testfailed) instead of 'success' in this case.

Comment 1

[Jonathan French (:jfrench)]

Attachment: marionetteUnitTests

Per screen grab.

Comment 2

[Ed Morley [:emorley]]

The job is marked as successful even though the log contains:

07:00:02 ERROR - TEST-START | test_errors.py TestAllExceptions.test_properties
07:00:02 ERROR - TEST-PASS | test_errors.py TestAllExceptions.test_properties | took 158ms
07:00:02 ERROR - TEST-START | test_errors.py TestExceptionType.test_cause_string
07:00:02 ERROR - TEST-PASS | test_errors.py TestExceptionType.test_cause_string | took 166ms
07:00:02 ERROR - TEST-START | test_errors.py TestExceptionType.test_cause_tuple
07:00:02 ERROR - TEST-PASS | test_errors.py TestExceptionType.test_cause_tuple | took 178ms
07:00:02 ERROR - TEST-START | test_errors.py TestExceptionType.test_construction
07:00:02 ERROR - TEST-PASS | test_errors.py TestExceptionType.test_construction | took 214ms
07:00:02 ERROR - TEST-START | test_errors.py TestExceptionType.test_defaults
07:00:02 ERROR - TEST-PASS | test_errors.py TestExceptionType.test_defaults | took 192ms
07:00:02 ERROR - TEST-START | test_errors.py TestExceptionType.test_str 

Treeherder displays the job result according to what buildbot tells it, and buildbot gets that from mozharness/marionette/...

Comment 3

[Chris Manchester (limited bugmail, email directly)]

:philor diagnosed this correctly in bug 1156070