1406663 - test-verify: "REFTEST TEST-UNEXPECTED-FAIL | | EXCEPTION: No tests to run" when running parallel reftests in a directory with few tests

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Technically a duplicate of bug 961234, but this will hit test-verify much more because it only runs the modified tests.

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=24cf61e5a04fb68ecc2b41bf3e858199783c5450&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable

Comment 1

[Intermittent Failures Robot]

4 failures in 824 pushes (0.005 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* autoland: 4

Platform breakdown:
* linux64: 4

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1406663&startday=2017-10-02&endday=2017-10-08&tree=all

Comment 2

[Geoff Brown [:gbrown] (pto Apr 11-18)]

TV tried to run the reftest layout/reftests/bugs/1404057-noref.html, with manifest entry

   != 1404057.html 1404057-noref.html

TV has a hack to avoid running <test>-ref.html. It looks like -noref, -noref1, -noref2 are also common conventions for reftest reference files.

Comment 3

[Geoff Brown [:gbrown] (pto Apr 11-18)]

I think bug 961234 is not relevant in this case...though it might be in another.

Comment 4

[Geoff Brown [:gbrown] (pto Apr 11-18)]

Attachment: handle more cases of reftest reference files

In the long term, I hope to make this more robust by upgrading our python ReftestManifest class. For now, we can avoid more failures by recognizing more cases: files ending in -noref.html, -noref1.html, etc are likely reference files.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=59820bc4db389e7f7d11eaee9c15406f438349c4

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/925b016fb5c0
Further improve test-verify behavior when reftest reference file is modified; r=jmaher

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/925b016fb5c0