Closed Bug 1587139 Opened Last month Closed Last month

Non-structured mozharness output parser does not check for unexpected test results effectively


(Testing :: General, defect, P1)

Version 3


(firefox71 fixed)

Tracking Status
firefox71 --- fixed


(Reporter: gbrown, Assigned: gbrown)




(1 file)

Tasks run with mozharness use an output parser to check for errors reported by the test harness. Many such tasks use the structured parser, while others -- notably android tests -- use the older, non-structured implementation, DesktopUnittestOutputParser.

DesktopUnittestOutputParser uses an ineffective check for failure strings like "UNEXPECTED-FAIL". As a result, some such failures can occur without causing a task failure: the task is "green" even though some tests have failed. This is a contributing factor to bug 1521640.

Specifically, DesktopUnittestOutputParser uses a regex match() to search each line of output; if the failure string is not at the beginning of the line, there is no match. For instance, Android crashtest failures begin with "REFTEST TEST-UNEXPECTED-FAIL | ..." so do not match "TEST-UNEXPECTED-FAIL". Let's search the whole line for known failure patterns.

Searching the whole line (s/match/search) appears to be effective.

demonstrates that Android crashtests fail due to existing assertion mismatch failures, independent of a fix for bug 1521640:

[task 2019-10-08T19:25:50.376Z] 19:25:50  WARNING -  REFTEST TEST-UNEXPECTED-FAIL | | assertion count 2 is more than expected 0 assertions
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - # TBPL WARNING #
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - setting return code to 1
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - The crashtest suite: crashtest ran with return status: WARNING
[task 2019-10-08T19:32:05.911Z] + exit 1
[taskcluster 2019-10-08 19:32:06.512Z] === Task Finished ===
[taskcluster 2019-10-08 19:32:11.746Z] Unsuccessful task run with exit code: 1 completed in 861.003 seconds

As far as I can see, the change has no adverse effects on other tasks:

When searching for "harness errors" as defined by the regexes at
use rather than re.match so that any logging prefixes do not defeat the match.

Priority: -- → P1
Pushed by
Improve mozharness search for test failure patterns; r=bc
Closed: Last month
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
You need to log in before you can comment on or make changes to this bug.