632754 - Annotate svg-as-an-image reftests that fail on d2d as "fails-if", not "random-if"

Status: RESOLVED FIXED

Description

[Daniel Holbert [:dholbert]]

Currently, the following tests are marked random-if(d2d):
> 63 random-if(d2d) == img-widthAndHeight-meet-1.html  img-widthAndHeight-meet-1-ref.html
> 64 random-if(d2d) == img-widthAndHeight-meet-2.html  img-widthAndHeight-meet-2-ref.html
> 65 random-if(d2d) == img-widthAndHeight-slice-1.html img-widthAndHeight-slice-1-ref.html
> 66 random-if(d2d) == img-widthAndHeight-slice-2.html img-widthAndHeight-slice-2-ref.html
> 67 
> 68 random-if(d2d) == img-height-meet-1.html   img-height-meet-1-ref.html
> 69 random-if(d2d) == img-height-meet-2.html   img-height-meet-2-ref.html
> 70 random-if(d2d) == img-height-slice-1.html  img-height-slice-1-ref.html
> 71 random-if(d2d) == img-height-slice-2.html  img-height-slice-2-ref.html
> 72 
> 73 random-if(d2d) == img-width-meet-1.html   img-width-meet-1-ref.html
> 74 random-if(d2d) == img-width-meet-2.html   img-width-meet-2-ref.html
> 75 random-if(d2d) == img-width-slice-1.html  img-width-slice-1-ref.html
> 76 random-if(d2d) == img-width-slice-2.html  img-width-slice-2-ref.html
http://mxr.mozilla.org/mozilla-central/source/layout/reftests/svg/as-image/reftest.list?force=1

I'm pretty sure they're all perma-fail (with a few px difference) on D2D-enabled windows, though -- not random.

Filing this bug on
  (a) verifying that the above is true on TryServer
  (b) applying s/random-if/fails-if/ (assuming that (a) succeeds)

Comment 1

[Daniel Holbert [:dholbert]]

I just gave a quick whack at (a), layered on top of my patches for bug 614649 -- we'll see how it turns out.

Tentative-patch: http://hg.mozilla.org/try/rev/866ed3a1ea1d
TryServer tbpl: http://tbpl.mozilla.org/?tree=MozillaTry&rev=575929d8cb0f

Comment 2

[Daniel Holbert [:dholbert]]

Turns out some of the tests seem to pass.  Pushed to try again with the fails-if/random-if(d2d) annotation completely removed for those tests, and it came up green:  http://tbpl.mozilla.org/?tree=MozillaTry&43f9c9e39e66

I just requested additional reftest runs for that build, to get a little more statistical significance. If they're green, too, then I think we can assume these tests are stable.

Comment 3

[Daniel Holbert [:dholbert]]

oops, I left out the "rev=" in the URL from comment 2.  Meant to say:
  http://tbpl.mozilla.org/?tree=MozillaTry&rev=43f9c9e39e66

Comment 4

[Daniel Holbert [:dholbert]]

Ok, cool - so I got 4 reftest runs each, on Win debug and Win opt, and they were all green (modulo a known randomorange in plugins code) - so I think it's safe to say that these tests all reliably pass or fail as marked in that TryServer push:  http://hg.mozilla.org/try/rev/315189c83a1c

Comment 5

[Daniel Holbert [:dholbert]]

Created attachment 511275 [details] [diff] [review]
fix: annotate tests as passing/failing based on TryServer performance

Comment 6

[Daniel Holbert [:dholbert]]

I don't think there's a bug tracking these D2D failures themselves, so I filed bug 633072 on that.

I added a comment with that bug number to all the "fails-if(d2d)" entries in as-image/reftest.list, and landed: http://hg.mozilla.org/mozilla-central/rev/f00a192644fd