Closed Bug 1530008 Opened 2 years ago Closed 2 years ago

Reftest analyzer doesn't work with local runs of WPT

Categories

(Testing :: Reftest, enhancement)

enhancement
Not set
normal

Tracking

(firefox67 fixed)

RESOLVED FIXED
mozilla67
Tracking Status
firefox67 --- fixed

People

(Reporter: gsnedders, Assigned: gsnedders)

Details

Attachments

(1 file)

It seems the reftest analyzer still doesn't work with output from WPT.

An example test for reftest-analyzer.xhtml's testsuite:

    { "name": "wpt reftests (Linux local)",
      "log": `SUITE-START | Running 1 tests
TEST-START | /css/css-backgrounds/border-image-6.html
TEST-UNEXPECTED-FAIL | /css/css-backgrounds/border-image-6.html | Testing http://web-platform.test:8000/css/css-backgrounds/border-image-6.html == http://web-platform.test:8000/css/css-backgrounds/border-image-6-ref.html
REFTEST   IMAGE 1 (TEST): data:image/png;base64,
REFTEST   IMAGE 2 (REFERENCE): data:image/png;base64,
TEST-INFO took 425ms
SUITE-END | took 2s`,
      "expected": { "pass": 0, "unexpected": 1, "random": 0, "skip": 0 },
    },

Bug 1362811 changed enough for WPT reftests to work with reftest-analyzer.xhtml in automation, due to the mozharness prefix; the conclusion of bug 1265582 was to add reftest-analyzer-structured.xhtml but that can't be used without a local copy due to Bug 1530002.

It's not clear what we should fork on to support output from mozlog; should we just support TEST-UNEXPECTED-FAIL etc. without any prefix whatsoever?

I'd be fine with support it with no prefix.

Assignee: nobody → geoffers+mozilla
Status: NEW → ASSIGNED
Pushed by btara@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/a351f6e4fb9c
make reftest-analyzer.xhtml work with local WPT output r=dbaron
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.