The 'fails' and 'fuzzy' annotations do not interact correctly

NEW
Unassigned

Status

defect
5 years ago
2 years ago

People

(Reporter: seth, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

When a reftest has both a 'fails' annotation and a 'fuzzy' annotation, fails should be evaluated with respect to the fuzzy annotation, not with respect to the underlying test. An example of the problem:

> fuzzy(5,999) fails-if(!cocoaWidget) != test.html ref.html

The intention here is to use the behavior of fuzzy with != to make sure that the maximum difference between test.html and ref.html is more than 5, and annotate that this test fails on non-OS-X platforms.

Unfortunately, that doesn't work, because the fails-if annotation ignores the fuzzy annotation. Instead, you get a TEST-UNEXPECTED-PASS if test.html and ref.html differ on some OS X platform, even if the maximum difference is less than 5.

There is actually no way to express this correctly with the reftest harness right now, as far as I can see. I've tried many possible ways of expressing this and they don't work.

This seems related, but different, to bugs like bug 720684.
See Also: → 1299325
You need to log in before you can comment on or make changes to this bug.