523864 - reftest harness should trap missing file exceptions and continue

Status: NEW

(Testing :: Reftest, defect)

Description

[Zack Weinberg (:zwol)]

Reftests currently do not deal well with a missing file:

If the file in a "load" test, or the first file in a "!=" or "==" test, is
missing, you get TEST-UNEXPECTED-FAIL || EXCEPTION: ... in the log and the test run is abandoned at that point.  It should detect NS_ERROR_FILE_NOT_FOUND and go on to the next test.

Worse, if the *second* file (the reference) in a != or == test is missing, the test *hangs* with the reftest window showing the "file not found" screen.  I'm not sure why there's a difference there.

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

It should definitely *always* be a test failure (even if marked "fails"), though.

Comment 2

[Zack Weinberg (:zwol)]

Agreed.