Talos test unexpected exit is not reflected on treeherder talos job

NEW
Unassigned

Status

defect
2 years ago
2 years ago

People

(Reporter: rwood, Unassigned)

Tracking

Version 3
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

If talos run_tests.py is exited due to an error, i.e. [1][2][3] the failure is not reflected on treeherder. For example, if one test in the talos suite is exited via sys.exit(), then on treeherder you'll just see a green talos test job but the affected test just won't be listed in the job results.

The treeherder job itself should be marked as failed if one of the talos tests in the suite failed to run because of an issue.

[1] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#156

[2] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#205

[3] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#280

I tried replacing one of the 'sys.exit'(s) with 'raise TalosError' but that doesn't reflect on treeherder either.
You need to log in before you can comment on or make changes to this bug.