1587139 - Non-structured mozharness output parser does not check for unexpected test results effectively

Status: RESOLVED FIXED

(Testing :: General, defect, P1)

Description

[Geoff Brown [:gbrown]]

Tasks run with mozharness use an output parser to check for errors reported by the test harness. Many such tasks use the structured parser, while others -- notably android tests -- use the older, non-structured implementation, DesktopUnittestOutputParser.

DesktopUnittestOutputParser uses an ineffective check for failure strings like "UNEXPECTED-FAIL". As a result, some such failures can occur without causing a task failure: the task is "green" even though some tests have failed. This is a contributing factor to bug 1521640.

Comment 1

[Geoff Brown [:gbrown]]

Specifically, DesktopUnittestOutputParser uses a regex match() to search each line of output; if the failure string is not at the beginning of the line, there is no match. For instance, Android crashtest failures begin with "REFTEST TEST-UNEXPECTED-FAIL | ..." so do not match "TEST-UNEXPECTED-FAIL". Let's search the whole line for known failure patterns.

Comment 2

[Geoff Brown [:gbrown]]

Searching the whole line (s/match/search) appears to be effective.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ea36b60e2c9cd2fd33628ef60d0571ce1e62df40

demonstrates that Android crashtests fail due to existing assertion mismatch failures, independent of a fix for bug 1521640:

[task 2019-10-08T19:25:50.376Z] 19:25:50  WARNING -  REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/base/crashtests/690247-1.html | assertion count 2 is more than expected 0 assertions
...
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - # TBPL WARNING #
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - setting return code to 1
[task 2019-10-08T19:32:03.778Z] 19:32:03  WARNING - The crashtest suite: crashtest ran with return status: WARNING
...
[task 2019-10-08T19:32:05.911Z] + exit 1
[taskcluster 2019-10-08 19:32:06.512Z] === Task Finished ===
[taskcluster 2019-10-08 19:32:11.746Z] Unsuccessful task run with exit code: 1 completed in 861.003 seconds

Comment 3

[Geoff Brown [:gbrown]]

As far as I can see, the change has no adverse effects on other tasks:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0c1564d5ac39bfb56649e1e08fc0f23bf5e4e34a

Comment 4

[Geoff Brown [:gbrown]]

Attachment: Bug 1587139 - Improve mozharness search for test failure patterns; r=

When searching for "harness errors" as defined by the regexes at
https://searchfox.org/mozilla-central/rev/5cb522c7baba24e55874809e0e206b001494c1e9/testing/mozharness/mozharness/mozilla/testing/errors.py#112
use re.search rather than re.match so that any logging prefixes do not defeat the match.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c5f0a88c97f8
Improve mozharness search for test failure patterns; r=bc

Comment 6

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/c5f0a88c97f8