1090627 - Disable failing reftests to get Linux opt e10s tests green

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: disable-reftests

Comment 1

[David Baron :dbaron:]

Why skip-if rather than fails-if?  skip-if should generally be used for crashes; fails-if should be used for failures, so when the failures are fixed we'll know to reenable the tests.

Comment 2

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

I can use fail-if for the gradient tests. The video tests are timing out so I think we want skip-if there.

Comment 3

[Matt Woodrow (:mattwoodrow)]

Comment on attachment 8513119 [details] [diff] [review]
disable-reftests

Review of attachment 8513119 [details] [diff] [review]:
-----------------------------------------------------------------

Yeah, make the gradient tests fails-if please :)

Comment 4

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/f2e96abcf610

Comment 5

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Made a stupid mistake:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a67645f84f65
https://hg.mozilla.org/integration/mozilla-inbound/rev/1aa71eb718f7

Comment 6

[Carsten Book [:Tomcat]]

sorry had to back 1aa71eb718f7 out for test failures like https://treeherder.mozilla.org/ui/logviewer.html#?job_id=3427216&repo=mozilla-inbound

Comment 7

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Worked around the b2g issue:
https://hg.mozilla.org/integration/mozilla-inbound/rev/af652f97938d

However, it looks like more tests are failing now.

Comment 8

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/af652f97938d

Comment 9

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: flip-test-plugin

Now that plugins are working, some reftests that were marked as failing are now passing. This is progress!

Comment 10

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Attachment: fix-fuzzy

There seems to be some sort of bad interaction here between fails-if and fuzzy. Somehow the fails-if annotation isn't having any effect. I still see:
 0:07.64 REFTEST TEST-UNEXPECTED-FAIL | file:///home/billm/moz/in3/layout/reftests/css-gradients/large-gradient-3.html | image comparison (==), max difference: 211, number of differing pixels: 800000

It looks like fuzzy overrides whatever state is set by fails-if. I worked around this using fuzzy-if. Is this the right way to do this?

Comment 11

[Matt Woodrow (:mattwoodrow)]

(In reply to Bill McCloskey (:billm) from comment #10)
> Created attachment 8515401 [details] [diff] [review]
> fix-fuzzy
> 
> There seems to be some sort of bad interaction here between fails-if and
> fuzzy. Somehow the fails-if annotation isn't having any effect. I still see:
>  0:07.64 REFTEST TEST-UNEXPECTED-FAIL |
> file:///home/billm/moz/in3/layout/reftests/css-gradients/large-gradient-3.
> html | image comparison (==), max difference: 211, number of differing
> pixels: 800000
> 
> It looks like fuzzy overrides whatever state is set by fails-if. I worked
> around this using fuzzy-if. Is this the right way to do this?

I think just swapping the order would work too, but that seems fine to me.

Comment 12

[David Baron :dbaron:]

Comment on attachment 8515401 [details] [diff] [review]
fix-fuzzy

Yeah, fuzzy is pretty broken.  Swapping the order would probably work, but the fails annotation (and others) doesn't fully override fuzzy.

Comment 13

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/799df20c9b98
https://hg.mozilla.org/integration/mozilla-inbound/rev/db4eb0dbd8e2

I want to leave this open until tests are actually green.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/799df20c9b98
https://hg.mozilla.org/mozilla-central/rev/db4eb0dbd8e2

Comment 15

[Bill McCloskey [inactive unless it's an emergency] (:billm)]

Tests are green and running.