Closed Bug 1195499 Opened 6 years ago Closed 6 years ago

[Win10 Reftest] Permafail - Aliasing differences between reference and test


(Testing :: Reftest, defect)

Not set


(firefox43 fixed)

Tracking Status
firefox43 --- fixed


(Reporter: Callek, Assigned: Callek)




(2 files)

From Bug 1193447:

(In reply to Bas Schouten (:bas.schouten) from comment #2)
> Then there's a bunch of very subtle anti-aliasing differences, those are
> most likely just improvements to Direct2D showing up and should be best
> dealt by fuzzing them.

I've volunteered to do this work for the worked-thin Graphics Team.
Notes for myself:

3:02 PM <Bas> Callek_cloud: Just FYI:, note the fuzzy-if, the first argument is a bool, the second argument is the max difference for a fuzzed pixel, and the third argument is how many pixels are allowed to have that difference.
3:03 PM <Callek_cloud> Bas: so should I specify the third argument to "whatever is the win10 failure" or is it likely to be a range or needs-higher
3:03 PM <Callek_cloud> Bas: aka I don't mind doing the tedious change myself for those :-)
3:04 PM <Callek_cloud> err and similar with second arg
3:04 PM <Callek_cloud> e.g. "what is the max difference" (or how do I calc it)
3:04 PM <Bas> Callek_cloud: Right, for all the ones that basically are these tiny edge differences of no more than 1 RGB value difference (i.e. max difference one and not like over more than a couple of dozen pixels), you just put in exactly the failure arguments in. The harness should tell you both max difference and amount of differing pixels.
3:07 PM <Bas> Callek_cloud: Right, yeah, so things like REFTEST TEST-UNEXPECTED-FAIL | file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/css-gradients/radial-shape-closest-side-1b.html | image comparison (==), max difference: 1, number of differing pixels: 4  are the ones you want to fuz with fuzzy-if(win10,1,4) for example :)
3:08 PM <Callek_cloud> Bas: \o/
3:08 PM <Callek_cloud> Bas: do you know for sure that fuzzy-if(win10,**) is the right connotation?
3:08 PM <Callek_cloud> or is "win10" an unknown from your perspective
3:08 PM <Bas> Callek_cloud: win10 was just a phony, I'm not sure if we have a way to detect win10
3:09 PM <Bas> Callek_cloud: Things like REFTEST TEST-UNEXPECTED-FAIL | file:///C:/slave/test/build/tests/reftest/tests/layout/reftests/svg/filters/css-filter-chains/long-chain.html | image comparison (==), max difference: 35, number of differing pixels: 13055  are too much difference, those you should defer to me for diagnosis.

example at:

And that condition is based roughly on the user agent, which thankfully reftest outputs actual debug output for us here!

00:39:39     INFO -  REFTEST INFO | Dumping JSON representation of sandbox
00:39:39     INFO -  REFTEST INFO | {"isDebugBuild":true,"xulRuntime":{"widgetToolkit":"windows","OS":"WINNT","XPCOMABI":"x86_64-msvc"},"smallScreen":false,"d2d":true,"azureQuartz":false,"azureSkia":false,"skiaContent":false,"azureSkiaGL":0,"contentSameGfxBackendAsCanvas":true,"layersGPUAccelerated":true,"layersOpenGL":false,"layersOMTC":true,"B2G":false,"B2GDT":false,"Android":false,"cocoaWidget":false,"gtkWidget":false,"qtWidget":false,"winWidget":true,"transparentScrollbars":false,"AddressSanitizer":false,"webrtc":true,"http":{"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:43.0) Gecko/20100101 Firefox/43.0","appName":"Mozilla","appVersion":"5.0","platform":"Windows","oscpu":"Windows NT 10.0; Win64; x64","misc":"rv:43.0"},"haveTestPlugin":true,"windowsDefaultTheme":true,"nativeThemePref":true,"prefs":{},"browserIsRemote":false,"Mulet":false,"asyncPan":false,"asyncZoom":false}
I went through annotations that exist for win8 and pushed to try for the changes relating to the following bugs.

due to Bug 850194 -- disable bug415761.sjs bug415761.ico  (already disabled on win8)

Matched the following annotations from w8:
 * Bug 850454
     == scroll-1.html scroll-1-ref.html 

 * Bug 849771 - "adjust annotations in layout/reftests/bugs for Win8"
     == 379316-2.html == 379316-2-ref.html

 * Bug 1128229 - Add fuzzy annotations to Android and Windows test failures caused by failure to call imgFrame::Optimize before the reftest snapshot is taken
     == 456219-1a.html 456219-1-ref.html
     == 456219-1b.html 456219-1-ref.html
     == 456219-1c.html 456219-1-ref.html

 * Bug 991046 - Add some fuzzy to the test for windows 8. 
     == 991046-1.html 991046-1-ref.html

 * Bug 1054838, part 5 - Mark some tests as now passing and add/change the fuzz on others post-mattwoodrow's Moz2Dification of gfxPattern 
     == radial-shape-closest-corner-1b.html radial-shape-closest-corner-1-ref.html

 * Bug 849772 - make certain layout/reftests/css-gradients tests fuzzy for Win8
     == radial-shape-closest-side-1a.html radial-shape-closest-side-1-ref.html
     == radial-shape-closest-side-1b.html radial-shape-closest-side-1-ref.html
     == radial-shape-closest-side-1c.html radial-shape-closest-side-1-ref.html

 * Bug 951541 - animateMotion-flattening-1.svg fails on Windows 8
     == animateMotion-flattening-1.svg lime.svg

...and pushed to try so I can check on state after-these tests run:
3:46 PM <Bas> So I think you can fuzz 784573 at least.
3:46 PM <Bas> and clipPath-and-shape-rendering-01.svg

So in addition to c#2's list of what I did we have the test for 784573 fuzzed and newly found:

* Bug 614840 - Intermittent failure in layout/reftests/svg/clipPath-and-shape-rendering-01.svg | image comparison (==)
    == clipPath-and-shape-rendering-01.svg clipPath-and-shape-rendering-01-ref.svg

* Bug 1024271 - Allow more fuzziness for the win64 test machines. r=dholbert
* ++
* Bug 937994 - Part 2: Mark some tests fuzzy, and change the discontinuity side of joins they expect. r=dholbert 
    == textLength-3.svg textLength-3-ref.svg

Try push at:

Based on
Attachment #8649606 - Flags: review?(bas)
(In reply to Justin Wood (:Callek) from comment #2)
> I went through annotations that exist for win8....

I think you also want to do this for a number of tests in editor/reftests/xul that are currently failing.
These got missed in my initial dxr searching, adding them as a separate patch (folding on landing is not out of the question if desired)

This is related to:

* Bug 847893 - adjust xul editor test annotations to include Win8
* &&
* Bug 623423 - Fix the editor XUL reftests for Windows XP compatibility
* &&
* Bug 574109 - failures in layout/reftests/editor/xul/ on windows 7 64-bit;
* &&
* (initial commit in) Bug 544543 - Add some tests for xul textboxes to make sure their basic behavior and appearance does not regress; r=roc 
    == autocomplete-1.xul autocomplete-ref.xul
    == emptyautocomplete-1.xul emptyautocomplete-ref.xul
    == number-3.xul number-ref.xul

* new to this bug - per jfkthame
    == 1177726-text-stroke-bounds.html 1177726-text-stroke-bounds-ref.html
    == line-break-suppression-1.html line-break-suppression-1-ref.html
    == line-break-suppression-2.html line-break-suppression-2-ref.html
    == ruby-reflow-1-transparentruby.html ruby-reflow-1-noruby.html
    == placeholder-6.html placeholder-overflow-ref.html
    == iframe-border-radius.html iframe-border-radius-ref.html

11:03 AM <jfkthame> note that 1115916 is fairly fuzzy already, so i'd be inclined to just adjust the fuzz amount, it's a test that can't currently achieve a perfect match
   for: == 1115916-1-vertical-metrics.html 1115916-1-vertical-metrics-ref.html

11:03 AM <jfkthame> the non-scaling-stroke one i'd be inclined to just fuzz, it's edge conditions
   for: == non-scaling-stroke-01.svg non-scaling-stroke-01-ref.svg

New Try Run (ontop of the last one):
Attachment #8649919 - Flags: review?(bas)
Comment on attachment 8649919 [details] [diff] [review]
[m-c] update fuzz list for xul as well

Review of attachment 8649919 [details] [diff] [review]:

Do you suppose it's worth creating a simple win10 bool in the harness to prevent all the cumbersome test conditions done right now?
Attachment #8649919 - Flags: review?(bas) → review+
Attachment #8649606 - Flags: review?(bas) → review+
You need to log in before you can comment on or make changes to this bug.