Hide ignored permafailures in cross-browser test suites like web-platform-tests (wpt)

NEW
Unassigned

Status

enhancement
2 years ago
2 years ago

People

(Reporter: aryx, Unassigned)

Tracking

(Blocks 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

• For the web-platform-tests (wpt), the expected test results are saved instead of unreliable tests getting disabled. These expected and ignored failures are part of the failure summary and the person classifying the failures has to find the real, unexpected failure by comparing the failure summary from the test run marked as failing with the one marked as passing.
    • Example: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=785572419acc82b2cbdcd2e24ca59fdbf5d7255f&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&filter-searchStr=web-platform-tests&selectedJob=149513922 with https://treeherder.mozilla.org/logviewer.html#?job_id=149513922&repo=mozilla-central
One option is for the test harness to suppress these from the log.

The other option is for Treeherder to start taking the mozharness "INFO - " prefix literally and treat the line as not an error. The Treeherder team would actually really like this approach, since we want to eventually get to the point where mozharness/... is the source of truth, and Treeherder doesn't need a long list of hardcoded regexes. However I imagine if we do this, there will be many other cases where the log level is incorrectly "INFO - " for a real failure - so there will be a lot of work to get this landable.

Do you have a preference for either of these?
Blocks: 778688
Flags: needinfo?(aryx.bugmail)
If the "INFO -" lines get removed, the "Assertion failure" ones also seem to get removed: https://treeherder.mozilla.org/logviewer.html#?job_id=153700646&repo=autoland&lineNumber=24080
> 01:04:53     INFO - PID 2996 | Assertion failure: !JS_IsDetachedArrayBufferObject(buffer), at z:/build/build/src/js/src/builtin/Stream.cpp:5141
Is there anything else which would get drop? Can we keep those with regexps and drop the other "INFO - " stuff?
Flags: needinfo?(aryx.bugmail)
Yeah that's the tricky thing - there are lots of instances that's aren't doing the right thing so would need fixing in the upstream harness/... first.

It sounds like the first option in comment 1 is the best way forwards - on which case this sounds like a WPT (/whatever wraps WPT) bug.
Component: Treeherder: Log Parsing & Classification → web-platform-tests
Product: Tree Management → Testing
Version: --- → unspecified
You need to log in before you can comment on or make changes to this bug.