Open Bug 914301 Opened 6 years ago Updated 6 years ago

Make the jit-test harness spit out useful errors when a test failures

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: RyanVM, Unassigned)

Details

We currently have 71 open bug for intermittent jit-test failures that we've hit on TBPL [1]. One major barrier we've hit in getting these fixed is that the harness doesn't print out any usable information to point to what failed. For example, below is a recent ASAN failure (soon to be filed):

https://tbpl.mozilla.org/php/getParsedLog.php?id=27588029&tree=B2g-Inbound

TEST-UNEXPECTED-FAIL | /builds/slave/b2g-in-l64-asan-00000000000000/build/js/src/jit-test/tests/ion/entryOverflowBailout.js | : ==20396==ABORTING

The above line is the entirety of the failure printed in the log, which obviously makes it pretty darn hard to figure out what went wrong. We need to improve the harness so that failures actually give developers some chance of determining what actually went wrong.

[1] https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure%2C%20&keywords_type=allwords&list_id=7868392&short_desc=jit-test&resolution=---&query_format=advanced&short_desc_type=allwordssubstr
The current code outputs the last line of output in general (be it stderr or out):

Making the change to output more is easy, the question is, what to output and when. I think in the failure case we should output all of stderr and the last line of stdout to be somewhat compatible with the previous behavior.

Does that make sense? :)
I have a patch that spills out stdout and stderr on failure, like this: 


TEST-UNEXPECTED-FAIL | /builds/slave/try-l64-asan-00000000000000000/build/js/src/jit-test/tests/TypedObjects/fuzz1.js | --baseline-eager --no-ti --no-fpu: err99
===> stdout <===
out1
out2
out3
out4
===> stderr <===
err96
err97
err98
err99


However, in the TBPL overview, only the first line will be visible (like it is now). You would have to open the parsed log to see all of the output.
That sounds great to me.
You need to log in before you can comment on or make changes to this bug.