1376288 - Talos test unexpected exit is not reflected on treeherder talos job

Status: NEW

Description

[Robert Wood [:rwood]]

If talos run_tests.py is exited due to an error, i.e. [1][2][3] the failure is not reflected on treeherder. For example, if one test in the talos suite is exited via sys.exit(), then on treeherder you'll just see a green talos test job but the affected test just won't be listed in the job results.

The treeherder job itself should be marked as failed if one of the talos tests in the suite failed to run because of an issue.

[1] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#156

[2] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#205

[3] http://searchfox.org/mozilla-central/rev/1b7b1ec949497e9fc2a9b9dfaccbe894ee2ad5ef/testing/talos/talos/run_tests.py#280

I tried replacing one of the 'sys.exit'(s) with 'raise TalosError' but that doesn't reflect on treeherder either.