Open Bug 454351 Opened 11 years ago Updated 9 years ago

An improved reftest analysis tool (alternative to reftest-analyzer.xhtml)

Categories

(Testing :: Reftest, enhancement)

enhancement
Not set

Tracking

(Not tracked)

People

(Reporter: zwol, Unassigned)

Details

Attachments

(2 files)

Attached file new tool
I've written an improved reftest analysis tool.  It's a Python script that you feed a log file to; it produces a directory full of image files plus an index.html.  You then load the index.html in your browser (note: almost certainly does not work in IE ;-).

Once you've done that you get a similar interface to the existing reftest-analyzer.xhtml, but (IMO) improved in four important ways.  First, it's much much faster, mainly because the "highlight differences" overlay image is precomputed by the Python script.  Second, the images are not cut off at the right or bottom; if they're too big for the window you get scroll bars, independent of the scroll bar for the test case list.  Third, the difference highlight is easier to grok at a glance, especially for small differences involving text.  Fourth, the list of failing test cases does not waste space on TODOs that you can't look at.

There are also some spiffy bells and whistles, like you can pipe the output of ./dist/bin/firefox -reftest directly to this tool, give it the --progress switch, and it'll announce failures as they happen.

Of course it could be better still (on my list are image zoom, showing TODOs (needs changes to the reftest runner), and being able to resize the list of failing test cases horizontally) but I think this is already good enough for others to play with.
For what it's worth, reftest-analyzer.xhtml was never intended to be a finished product -- really more like a proof of concept that I never had time to polish -- and a bunch of these improvements are relatively simple to make there (although speeding up SVG filters is a bit harder).  I intended to make the list on the left filterable and independently scrollable, just never got around to it; see http://hg.mozilla.org/mozilla-central/annotate/af9ae0314398/layout/tools/reftest/reftest-analyzer.xhtml#l40

That said, I'm happy to have this one in the tree as well.  (Haven't looked at the code yet, though.)
This output file doesn't get completely translated, only three tests show up in the new tool's HTML output.  using the old layout/tools/reftest-to-html.pl script, I get everything.

The last test parsed by the new tool is a failing test.  Is it possible that the tool stops on failure for some reason?

I ran it by feeding it the attached output file rather than piping the output to the analyzer.
(In reply to comment #2)
> 
> This output file doesn't get completely translated, only three tests show up in
> the new tool's HTML output.  using the old layout/tools/reftest-to-html.pl
> script, I get everything.

We discussed this on IRC - for the record, the tool as presently coded intentionally presents only the failures, but I intend to add an option to show successes and todos as well.
dbaron: I was aware of the note in reftest-analyzer.xhtml but found the SVG slowness enough of a problem that I wanted to write something that precomputed the images anyway.  You could probably steal most of my HTML and Javascript if you wanted.
Component: Testing → Reftest
Product: Core → Testing
Version: Trunk → unspecified
QA Contact: testing → reftest
I no longer work for Mozilla, I am deassigning myself from bugs I have no intention of working on as a volunteer.
Assignee: zackw → nobody
Status: ASSIGNED → NEW
Summary: An improved reftest analysis tool → An improved reftest analysis tool (alternative to reftest-analyzer.xhtml)
You need to log in before you can comment on or make changes to this bug.