372196 - reftests reports should provide better image comparison UI

Status: RESOLVED FIXED

(Testing :: Reftest, defect)

Description

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Reftest reports (as generated by clean-reftest-report.pl) should provide a better way of seeing the image differences.  We could probably do this using canvas getImageData (and generate an image that's only opaque where there are differences), plus some overlaying of images with opacity.

Comment 1

[Boris Zbarsky [:bzbarsky]]

Note to self: for now I can use http://blog.mozilla.com/rob-sayre/2007/02/28/finding-the-difference-between-reftest-images

Comment 2

[Justin Dolske [:Dolske]]

I did a quick test, and just *calling* the canvas's .getImageData() was really slow... On my laptop, reftests were normally taking roughly 1 second per test. I then hacked reftest.js's IFrameToKey() as follows:

    //return gCanvas.toDataURL();
    var ignored = ctx.getImageData(0, 0, gCanvas.width, gCanvas.height);
    return "foo";

Test performance then dropped to an abysmal 10-15 seconds per test. [And, yes, just returning "foo" without calling .getImageData() at all is lightning fast!]

I don't know if the slowness is just overhead due to schlepping bits into a JS array, or if something deeper down is slow. The new Cairo version is supposed to have some OS X performance improvements, maybe that will help? However... I recently converted some code in my APNG editor to use atob() instead of a JS implementation [to write a data: PNG to a file], and it was enormously faster. That leads me to believe any kind of pixel pushing in JS is simply going to be slow.

I was thinking it should be faster to have a scriptable C++ component implement an interface like:

    boolean areEqual(in nsIDOMCanvasRenderingContext2D aCanvas1,
                     in nsIDOMCanvasRenderingContext2D aCanvas2);

..and have the code do a memcmp() or whatever between the two.

To see the differences, it might as well implement another interface to compare the the two canvases, and modify a 3rd canvas with the differences. For example, the caller could clone the reference canvas to a temporary canvas with low opacity, and then the comparison code would flip pixels there to bright red wherever there's a difference. Or maybe a greyscale difference map would be better?

Comment 3

[Boris Zbarsky [:bzbarsky]]

Justin, on OSX you should be able to use Shark to tell you exactly where the time is spent....  But yes, I suspect doing this in JS in a performant way would be hard.  Start with the fact that for an 800x600 image GetImageData first allocates a 1.6MB chunk of memory, then allocates a 6.4MB chunk of memory, then creates a JS array with 480,000 entries, 

For what it's worth, I tried profiling this a bit on Linux, and I see 90+% of the GetImageData() being used by JS_NewArrayObject.  Mostly because SetArrayElement takes a bunch of time (between js_LookupProperty, array_addProperty, ChangeScope, GetPropertyTreeChild, js_SearchScope, etc).

So yeah, sounds like at this point just getting the data into JS is the expensive part.

Comment 4

[Adam Guthrie]

Vlad has a C++ impl. of CompareCanvas that's integrated into the reftest harness over in bug 387132, FWIW.

Comment 5

[Brian Crowder]

Marking fixed in response to comment #4 and the existence of the reftest analysis tool, please reopen if I am mistaken.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

There's also layout/tools/reftest/reftest-analyzer.xhtml