1256276 - Integrate reftest-analyzer directly into the html report

Status: NEW

(Testing :: Reftest, defect)

Description

[Shing Lyu [:shinglyu]]

We have this tool called Reftest analyzer (http://people.mozilla.org/~tmielczarek/reftest-analyzer.xhtml), which can highlight the difference between two reftest screenshots.

Once Bug 1253219 lands, we'll have both screenshots in the html report, but it's still hard to analyze the difference. It would be nice to include the difference image into the html report.

Comment 1

[Shing Lyu [:shinglyu]]

I'm planning to do a image diff in the html report code. But I'm not sure if it's the responsibility of the report formatter or not? Should we create the image diff in the test runner so all the report format can have that image? For example we can have base64 encoded version in plain text log and have <img> in html log.

Could you kindly give me some advice?

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

It's not clear to me what the HTML report is for.  I've never used it; I just use reftest-analyzer.

I don't think we want the reftest harness to do the image diffing, though, since that's expensive and will slow it down considerably.  It should be part of post-processing that happens only when people want to look at the differences.

Comment 3

[Shing Lyu [:shinglyu]]

By running "./mach reftest --log-html mylog.html". You get a html report like this:

https://bug1253219.bmoattachments.org/attachment.cgi?id=8726156

As you can see there are screenshots included (which is enabled in bug 1253219). So perhaps we can include the image diffing into the html log formatter code. This way you don't have to copy-paste the log into reftest-analyzer, you just run "./mach reftest --log-html mylog.html" and open mylog.html to see the screenshots and diff.

Comment 4

[Shing Lyu [:shinglyu]]

Attachment: Add reftest-analyzer's output to html report for easier debugging

I just copied and pasted reftest-analyzer's output, not sure how I can reuse reftest-analyzer.xhtml itself

Comment 5

[Shing Lyu [:shinglyu]]

Attachment: example_output.html

This is what you got after running "./mach reftest --log-html example_output.html <test file>"

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Comment on attachment 8734672 [details] [diff] [review]
Add reftest-analyzer's output to html report for easier debugging

Clearing stale feedback flag to get this bug out of the reftest triage dashboard.

Is this html report still a thing? If so then the patch you attached looks reasonable to me. Sure it copies some of the svg from reftest-analyzer but it doesn't matter so much if they get out of sync, since they are independent features.