1422965 - Hide ignored permafailures in cross-browser test suites like web-platform-tests (wpt)

Status: NEW

(Testing :: web-platform-tests, enhancement)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

• For the web-platform-tests (wpt), the expected test results are saved instead of unreliable tests getting disabled. These expected and ignored failures are part of the failure summary and the person classifying the failures has to find the real, unexpected failure by comparing the failure summary from the test run marked as failing with the one marked as passing.
    • Example: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=785572419acc82b2cbdcd2e24ca59fdbf5d7255f&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=web-platform-tests&selectedJob=149513922 with https://treeherder.mozilla.org/logviewer.html#?job_id=149513922&repo=mozilla-central

Comment 1

[Ed Morley [:emorley]]

One option is for the test harness to suppress these from the log.

The other option is for Treeherder to start taking the mozharness "INFO - " prefix literally and treat the line as not an error. The Treeherder team would actually really like this approach, since we want to eventually get to the point where mozharness/... is the source of truth, and Treeherder doesn't need a long list of hardcoded regexes. However I imagine if we do this, there will be many other cases where the log level is incorrectly "INFO - " for a real failure - so there will be a lot of work to get this landable.

Do you have a preference for either of these?

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

If the "INFO -" lines get removed, the "Assertion failure" ones also seem to get removed: https://treeherder.mozilla.org/logviewer.html#?job_id=153700646&repo=autoland&lineNumber=24080
> 01:04:53     INFO - PID 2996 | Assertion failure: !JS_IsDetachedArrayBufferObject(buffer), at z:/build/build/src/js/src/builtin/Stream.cpp:5141
Is there anything else which would get drop? Can we keep those with regexps and drop the other "INFO - " stuff?

Comment 3

[Ed Morley [:emorley]]

Yeah that's the tricky thing - there are lots of instances that's aren't doing the right thing so would need fixing in the upstream harness/... first.

It sounds like the first option in comment 1 is the best way forwards - on which case this sounds like a WPT (/whatever wraps WPT) bug.