Open Bug 1256276 Opened 4 years ago Updated 3 years ago

Integrate reftest-analyzer directly into the html report

Categories

(Testing :: Reftest, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: shinglyu, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(2 files)

We have this tool called Reftest analyzer (http://people.mozilla.org/~tmielczarek/reftest-analyzer.xhtml), which can highlight the difference between two reftest screenshots.

Once Bug 1253219 lands, we'll have both screenshots in the html report, but it's still hard to analyze the difference. It would be nice to include the difference image into the html report.
I'm planning to do a image diff in the html report code. But I'm not sure if it's the responsibility of the report formatter or not? Should we create the image diff in the test runner so all the report format can have that image? For example we can have base64 encoded version in plain text log and have <img> in html log.

Could you kindly give me some advice?
Flags: needinfo?(dbaron)
It's not clear to me what the HTML report is for.  I've never used it; I just use reftest-analyzer.

I don't think we want the reftest harness to do the image diffing, though, since that's expensive and will slow it down considerably.  It should be part of post-processing that happens only when people want to look at the differences.
Flags: needinfo?(dbaron)
Summary: Integrate reftest-ayalyzer directly into the html report → Integrate reftest-analyzer directly into the html report
By running "./mach reftest --log-html mylog.html". You get a html report like this:

https://bug1253219.bmoattachments.org/attachment.cgi?id=8726156

As you can see there are screenshots included (which is enabled in bug 1253219). So perhaps we can include the image diffing into the html log formatter code. This way you don't have to copy-paste the log into reftest-analyzer, you just run "./mach reftest --log-html mylog.html" and open mylog.html to see the screenshots and diff.
I just copied and pasted reftest-analyzer's output, not sure how I can reuse reftest-analyzer.xhtml itself
Attachment #8734672 - Flags: feedback?(dbaron)
Attached file example_output.html
This is what you got after running "./mach reftest --log-html example_output.html <test file>"
Blocks: 1287026
You need to log in before you can comment on or make changes to this bug.