Open Bug 1453074 Opened 5 years ago Updated 6 months ago

Reftest failures can be missed if the output is split across multiple lines

Categories

(Testing :: Reftest, defect)

Product:
Testing
Component:
Reftest
Version:
59 Branch
Type:
defect

Tracking

(Not tracked)

People

(Reporter: kats, Unassigned)

Details

See https://treeherder.mozilla.org/logviewer.html#?job_id=172906991&repo=try - this job is passing, but if you open the raw log and search for "UnexpectedPass" you'll see that one of the reftests is failing with an unexpected-pass result. But the json "strucutured output" is broken across multiple lines with DOMWINDOW spew in between, and presumably that's why it doesn't get picked up by whatever is parsing the log for results.

The try push is at https://treeherder.mozilla.org/#/jobs?repo=try&revision=6dd6c81630bf4e1cdf060012e60e49526e52464f - all the Linux R3 things should be orange but some of them are green because of this issue.
Component: General → Reftest
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.