1550586 - Better failure message when test harness summary missing

Status: NEW

(Testing :: General, defect, P3)

Description

[Geoff Brown [:gbrown]]

See bug 1550577 for inspiration.

Tests may not complete for a variety of reasons, all requiring task failure. Android is particularly prone to this, since memory hungry tests can exceed the Android OS oom thresholds, triggering os-initiated shutdown of processes: no crash to report, no test failure in logs, just a suddenly missing test app.

1. Can we distinguish "No tests run" from "Test summary not found"?

2. For "Test summary not found", would it be advantageous to report "Test summary not found. Last test seen %s"? For cases like bug 1550577, it certainly would be better to file a separate bug. But if some failure hits various tests at random, this could produce N separate bugs with failure counts below triage thresholds, instead of a large N failure count in one bug.

Also, consider:

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=245082736&repo=mozilla-central&lineNumber=29955

3. Can we avoid "No tests run" if a crash report was generated? Better to file a bug against the crash report.

Comment 1

[Geoff Brown [:gbrown]]

This is more difficult than I anticipated, since the error is generated in the log parser.

A couple of additional thoughts, hopefully for follow-up one day:

• Consider simply shortening the message to "Test summary not found", because really that is the condition that generates the message.
• Include self.crashed (and self.leaked?) in the condition at https://searchfox.org/mozilla-central/rev/7556a400affa9eb99e522d2d17c40689fa23a729/testing/mozharness/mozharness/mozilla/testing/unittest.py#206: Emphasize the real failure cause as much as possible. I struggled to demonstrate an improvement with this though.