Closed Bug 1587139 Opened Last month Closed Last month

Non-structured mozharness output parser does not check for unexpected test results effectively

Categories

(Testing :: General, defect, P1)

Version 3
defect

Tracking

(firefox71 fixed)

RESOLVED FIXED
mozilla71
Tracking Status
firefox71 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(1 file)

Tasks run with mozharness use an output parser to check for errors reported by the test harness. Many such tasks use the structured parser, while others -- notably android tests -- use the older, non-structured implementation, DesktopUnittestOutputParser.

DesktopUnittestOutputParser uses an ineffective check for failure strings like "UNEXPECTED-FAIL". As a result, some such failures can occur without causing a task failure: the task is "green" even though some tests have failed. This is a contributing factor to bug 1521640.

Specifically, DesktopUnittestOutputParser uses a regex match() to search each line of output; if the failure string is not at the beginning of the line, there is no match. For instance, Android crashtest failures begin with "REFTEST TEST-UNEXPECTED-FAIL | ..." so do not match "TEST-UNEXPECTED-FAIL". Let's search the whole line for known failure patterns.

Searching the whole line (s/match/search) appears to be effective.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ea36b60e2c9cd2fd33628ef60d0571ce1e62df40

demonstrates that Android crashtests fail due to existing assertion mismatch failures, independent of a fix for bug 1521640:

[task 2019-10-08T19:25:50.376Z] 19:25:50  WARNING -  REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/base/crashtests/690247-1.html | assertion count 2 is more than expected 0 assertions
...
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - # TBPL WARNING #
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - setting return code to 1
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - The crashtest suite: crashtest ran with return status: WARNING
...
[task 2019-10-08T19:32:05.911Z] + exit 1
[taskcluster 2019-10-08 19:32:06.512Z] === Task Finished ===
[taskcluster 2019-10-08 19:32:11.746Z] Unsuccessful task run with exit code: 1 completed in 861.003 seconds

As far as I can see, the change has no adverse effects on other tasks:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0c1564d5ac39bfb56649e1e08fc0f23bf5e4e34a

When searching for "harness errors" as defined by the regexes at
https://searchfox.org/mozilla-central/rev/5cb522c7baba24e55874809e0e206b001494c1e9/testing/mozharness/mozharness/mozilla/testing/errors.py#112
use re.search rather than re.match so that any logging prefixes do not defeat the match.

Priority: -- → P1
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c5f0a88c97f8
Improve mozharness search for test failure patterns; r=bc
Status: NEW → RESOLVED
Closed: Last month
Resolution: --- → FIXED
Target Milestone: --- → mozilla71
You need to log in before you can comment on or make changes to this bug.