web-platform-tests reftest runner gives output that doesn't work with our reftest analysis tools (e.g., reftest-analyzer)

RESOLVED FIXED in Firefox 52

Status

defect
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: dbaron, Unassigned)

Tracking

(Blocks 1 bug)

Trunk
mozilla52
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox48 affected, firefox52 fixed)

Details

Attachments

(2 attachments, 1 obsolete attachment)

If I run:
  ./mach web-platform-tests --test-types reftest
I get failure messages that are in a format that isn't compatible with the various tools we have such as reftest-analyzer (layout/tools/reftest/reftest-analyzer.xhtml), which are extremely useful for understanding the actual failures in reftests.

The output should match the format we're used to for reftest output so that our tools work.
http://hoppipolla.co.uk/410/reftest-analyser-structured.xhtml works on the structured log (it will also work on the structured log generated by reftest harness reftests). Is that an acceptable solution.
Could you merge that into the in-tree copy so that other people can make improvements to it?
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment on attachment 8795510 [details]
Bug 1265582 - Add a copy of reftest analyzer for working with structured log files,

https://reviewboard.mozilla.org/r/81550/#review80118

Could you post a version of this that records it as an hg copy from the original file?

That both:
 - makes it easier to review
 - preserves history and blame
Attachment #8795510 - Flags: review?(dbaron) → review-

Comment 6

3 years ago
mozreview-review
Comment on attachment 8795511 [details]
Bug 1265582 - Fix reftest structured logs to match wpt,

https://reviewboard.mozilla.org/r/81552/#review80276

Will this still work with the old reftest-analyzer? If not then we should probably update the treeherder link and post to the newsgroups first. If so are there plans to deprecate the old analyzer, or are we indefinitely punting on that?
Attachment #8795511 - Flags: review?(ahalberstadt) → review+
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment on attachment 8795793 [details]
Bug 1265582 - Create a copy of reftest analyzer that will be used for structured log files,

https://reviewboard.mozilla.org/r/81726/#review80386

This is still not an hg copy.  Running "hg copy -A [SOURCE] [DEST]" (see "hg help copy") should fix that after-the-fact.
Attachment #8795793 - Flags: review?(dbaron) → review-
Comment on attachment 8795510 [details]
Bug 1265582 - Add a copy of reftest analyzer for working with structured log files,

https://reviewboard.mozilla.org/r/81550/#review80388

The actual changes look fine here, but this should be in the same revision as the "hg copy".
Attachment #8795510 - Flags: review?(dbaron) → review-

Comment 13

3 years ago
mozreview-review-reply
Comment on attachment 8795510 [details]
Bug 1265582 - Add a copy of reftest analyzer for working with structured log files,

https://reviewboard.mozilla.org/r/81550/#review80388

Sure, that's just a pain to achieve because it seems git cinnabar doesn't preserve that metadata. But I am *slowly* updating my hg clone.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Attachment #8795793 - Attachment is obsolete: true
Comment on attachment 8795510 [details]
Bug 1265582 - Add a copy of reftest analyzer for working with structured log files,

https://reviewboard.mozilla.org/r/81550/#review80684
Attachment #8795510 - Flags: review?(dbaron) → review+

Comment 17

3 years ago
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/b888d1d778b8
Add a copy of reftest analyzer for working with structured log files, r=dbaron
https://hg.mozilla.org/integration/autoland/rev/22b1e96cfd05
Fix reftest structured logs to match wpt, r=ahal

Comment 18

3 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/b888d1d778b8
https://hg.mozilla.org/mozilla-central/rev/22b1e96cfd05
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla52
You need to log in before you can comment on or make changes to this bug.