1530008 - Reftest analyzer doesn't work with local runs of WPT

Status: RESOLVED FIXED

(Testing :: Reftest, enhancement)

Description

[Sam Sneddon [:gsnedders]]

It seems the reftest analyzer still doesn't work with output from WPT.

An example test for reftest-analyzer.xhtml's testsuite:

    { "name": "wpt reftests (Linux local)",
      "log": `SUITE-START | Running 1 tests
TEST-START | /css/css-backgrounds/border-image-6.html
TEST-UNEXPECTED-FAIL | /css/css-backgrounds/border-image-6.html | Testing http://web-platform.test:8000/css/css-backgrounds/border-image-6.html == http://web-platform.test:8000/css/css-backgrounds/border-image-6-ref.html
REFTEST   IMAGE 1 (TEST): data:image/png;base64,
REFTEST   IMAGE 2 (REFERENCE): data:image/png;base64,
TEST-INFO took 425ms
SUITE-END | took 2s`,
      "expected": { "pass": 0, "unexpected": 1, "random": 0, "skip": 0 },
    },

Bug 1362811 changed enough for WPT reftests to work with reftest-analyzer.xhtml in automation, due to the mozharness prefix; the conclusion of bug 1265582 was to add reftest-analyzer-structured.xhtml but that can't be used without a local copy due to Bug 1530002.

It's not clear what we should fork on to support output from mozlog; should we just support TEST-UNEXPECTED-FAIL etc. without any prefix whatsoever?

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I'd be fine with support it with no prefix.

Comment 2

[Sam Sneddon [:gsnedders]]

Attachment: Bug 1530008 - make reftest-analyzer.xhtml work with local WPT output

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a351f6e4fb9c
make reftest-analyzer.xhtml work with local WPT output r=dbaron

Comment 4

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/a351f6e4fb9c