Open Bug 1376288 Opened 3 years ago Updated 3 months ago

Talos test unexpected exit is not reflected on treeherder talos job

Categories

(Testing :: Talos, defect, P3)

Version 3
defect

Tracking

(Not tracked)

People

(Reporter: rwood, Unassigned)

Details

If talos run_tests.py is exited due to an error, i.e. [1][2][3] the failure is not reflected on treeherder. For example, if one test in the talos suite is exited via sys.exit(), then on treeherder you'll just see a green talos test job but the affected test just won't be listed in the job results.

The treeherder job itself should be marked as failed if one of the talos tests in the suite failed to run because of an issue.

[1] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#156

[2] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#205

[3] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#280

I tried replacing one of the 'sys.exit'(s) with 'raise TalosError' but that doesn't reflect on treeherder either.
Priority: -- → P2
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.