1057526 - The 'fails' and 'fuzzy' annotations do not interact correctly

Status: NEW

(Testing :: Reftest, defect)

Description

[Seth Fowler [:seth] [:s2h]]

When a reftest has both a 'fails' annotation and a 'fuzzy' annotation, fails should be evaluated with respect to the fuzzy annotation, not with respect to the underlying test. An example of the problem:

> fuzzy(5,999) fails-if(!cocoaWidget) != test.html ref.html

The intention here is to use the behavior of fuzzy with != to make sure that the maximum difference between test.html and ref.html is more than 5, and annotate that this test fails on non-OS-X platforms.

Unfortunately, that doesn't work, because the fails-if annotation ignores the fuzzy annotation. Instead, you get a TEST-UNEXPECTED-PASS if test.html and ref.html differ on some OS X platform, even if the maximum difference is less than 5.

There is actually no way to express this correctly with the reftest harness right now, as far as I can see. I've tried many possible ways of expressing this and they don't work.

This seems related, but different, to bugs like bug 720684.