Closed Bug 1090627 Opened 5 years ago Closed 5 years ago

Disable failing reftests to get Linux opt e10s tests green

Categories

(Testing :: Reftest, defect)

x86_64
Linux
defect
Not set

Tracking

(e10s+)

RESOLVED FIXED
Tracking Status
e10s + ---

People

(Reporter: billm, Assigned: billm)

References

(Blocks 1 open bug)

Details

Attachments

(3 files)

Attached patch disable-reftestsSplinter Review
No description provided.
Attachment #8513119 - Flags: review?(matt.woodrow)
Why skip-if rather than fails-if?  skip-if should generally be used for crashes; fails-if should be used for failures, so when the failures are fixed we'll know to reenable the tests.
I can use fail-if for the gradient tests. The video tests are timing out so I think we want skip-if there.
Comment on attachment 8513119 [details] [diff] [review]
disable-reftests

Review of attachment 8513119 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, make the gradient tests fails-if please :)
Attachment #8513119 - Flags: review?(matt.woodrow) → review+
Worked around the b2g issue:
https://hg.mozilla.org/integration/mozilla-inbound/rev/af652f97938d

However, it looks like more tests are failing now.
Keywords: leave-open
Attached patch flip-test-pluginSplinter Review
Now that plugins are working, some reftests that were marked as failing are now passing. This is progress!
Attachment #8515340 - Flags: review?(roc)
Attached patch fix-fuzzySplinter Review
There seems to be some sort of bad interaction here between fails-if and fuzzy. Somehow the fails-if annotation isn't having any effect. I still see:
 0:07.64 REFTEST TEST-UNEXPECTED-FAIL | file:///home/billm/moz/in3/layout/reftests/css-gradients/large-gradient-3.html | image comparison (==), max difference: 211, number of differing pixels: 800000

It looks like fuzzy overrides whatever state is set by fails-if. I worked around this using fuzzy-if. Is this the right way to do this?
Attachment #8515401 - Flags: review?(dbaron)
(In reply to Bill McCloskey (:billm) from comment #10)
> Created attachment 8515401 [details] [diff] [review]
> fix-fuzzy
> 
> There seems to be some sort of bad interaction here between fails-if and
> fuzzy. Somehow the fails-if annotation isn't having any effect. I still see:
>  0:07.64 REFTEST TEST-UNEXPECTED-FAIL |
> file:///home/billm/moz/in3/layout/reftests/css-gradients/large-gradient-3.
> html | image comparison (==), max difference: 211, number of differing
> pixels: 800000
> 
> It looks like fuzzy overrides whatever state is set by fails-if. I worked
> around this using fuzzy-if. Is this the right way to do this?

I think just swapping the order would work too, but that seems fine to me.
Comment on attachment 8515401 [details] [diff] [review]
fix-fuzzy

Yeah, fuzzy is pretty broken.  Swapping the order would probably work, but the fails annotation (and others) doesn't fully override fuzzy.
Attachment #8515401 - Flags: review?(dbaron) → review+
Tests are green and running.
Status: NEW → RESOLVED
Closed: 5 years ago
Keywords: leave-open
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.