Open Bug 1057526 Opened 10 years ago Updated 2 years ago

The 'fails' and 'fuzzy' annotations do not interact correctly


(Testing :: Reftest, defect)



(Not tracked)


(Reporter: seth, Unassigned)



When a reftest has both a 'fails' annotation and a 'fuzzy' annotation, fails should be evaluated with respect to the fuzzy annotation, not with respect to the underlying test. An example of the problem:

> fuzzy(5,999) fails-if(!cocoaWidget) != test.html ref.html

The intention here is to use the behavior of fuzzy with != to make sure that the maximum difference between test.html and ref.html is more than 5, and annotate that this test fails on non-OS-X platforms.

Unfortunately, that doesn't work, because the fails-if annotation ignores the fuzzy annotation. Instead, you get a TEST-UNEXPECTED-PASS if test.html and ref.html differ on some OS X platform, even if the maximum difference is less than 5.

There is actually no way to express this correctly with the reftest harness right now, as far as I can see. I've tried many possible ways of expressing this and they don't work.

This seems related, but different, to bugs like bug 720684.
See Also: → 1299325
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.