Open Bug 1057526 Opened 7 years ago Updated 4 years ago

The 'fails' and 'fuzzy' annotations do not interact correctly

Categories

(Testing :: Reftest, defect)

defect
Not set
normal

Tracking

(Not tracked)

People

(Reporter: seth, Unassigned)

References

Details

When a reftest has both a 'fails' annotation and a 'fuzzy' annotation, fails should be evaluated with respect to the fuzzy annotation, not with respect to the underlying test. An example of the problem:

> fuzzy(5,999) fails-if(!cocoaWidget) != test.html ref.html

The intention here is to use the behavior of fuzzy with != to make sure that the maximum difference between test.html and ref.html is more than 5, and annotate that this test fails on non-OS-X platforms.

Unfortunately, that doesn't work, because the fails-if annotation ignores the fuzzy annotation. Instead, you get a TEST-UNEXPECTED-PASS if test.html and ref.html differ on some OS X platform, even if the maximum difference is less than 5.

There is actually no way to express this correctly with the reftest harness right now, as far as I can see. I've tried many possible ways of expressing this and they don't work.

This seems related, but different, to bugs like bug 720684.
You need to log in before you can comment on or make changes to this bug.