Closed Bug 1563358 Opened 5 months ago Closed 4 months ago

Android web-platform-tests-reftests logs don't report maxDifference or totalPixels differing, alongside reftest failures

Categories

(Testing :: web-platform-tests, defect, P2)

Version 3
defect

Tracking

(firefox70 fixed)

RESOLVED FIXED
mozilla70
Tracking Status
firefox70 --- fixed

People

(Reporter: dholbert, Assigned: jgraham)

Details

Attachments

(1 file)

In e.g. this Try run with wpt reftest failures...
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ec7224ebb3e39351e49a900442f99804750238e

...the test failures look like this on most platforms:

INFO - TEST-START | /css/css-contain/contain-paint-clip-015.html
[...]
INFO - PID 8580 | 1562178423597	Marionette	INFO	No differences allowed
INFO - PID 8580 | 1562178423598	Marionette	INFO	Found 332 pixels different, maximum difference per channel 98
INFO - TEST-UNEXPECTED-FAIL | /css/css-contain/contain-paint-clip-015.html | [...]

But on android, that "No differences allowed" / "Found 332 pixels different, maximum difference per channel 98" text is entirely missing. On that platform, there's nothing between the TEST-START and TEST-UNEXPECTED-FAIL lines. This makes it hard to know the right values to use for fuzzy annotations on that platform.

If possible, it'd be great to get the "$X differences allowed" and "$Y pixels-different $Z max-difference" values reported in that log.

(It looks like this logging is present in the "logcat" artifact for these runs, so it's there -- it's just a bit buried.)

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(james)

So basically it seems like the marionette logs end up in the logcat (which is a different artifact) rather than the test output. Which means this is available but is inconvenient to find.

majazf: Can you look at this? I think one solution would be to add this information to the failure message so that it's sent back to the harness and printed there rather than being printed in the marionette logging.

Flags: needinfo?(james) → needinfo?(mjzffr)
Priority: -- → P2
Assignee: nobody → mjzffr
Flags: needinfo?(mjzffr)

We agreed that James will fix this as part of other Android wpt stuff he's going to work on soon.

Assignee: mjzffr → james

Instead of reporting the pixel differences in a logger info message,
put it in the text that's returned to the harness. This has a notable
advantage on android where this will cause it to be logged as part of
the harness logs rather than ending up in the logcat from the
device. It also makes these messages more accessible in other
consumers of the logs e.g. wpt.fyi.

Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/f5f5cbb377ac
Report pixel differences as part of message in wpt reftests, r=maja_zf

Backed out changeset f5f5cbb377ac (bug 1563358) for eslint failure at /marionette/reftest.js on a CLOSED TREE.

Backout link: https://hg.mozilla.org/integration/autoland/rev/2e724ef1c10286e537104d6d812480611e680c81

**Push with failures:**https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=f5f5cbb377ac92a59c26a0f1de8b971ef878dee3&selectedJob=259261764

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=259261764&repo=autoland&lineNumber=234

Log snippet:

[task 2019-07-31T21:01:59.998Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-07-31T21:01:59.998Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-07-31T21:01:59.998Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-07-31T21:01:59.998Z]
[task 2019-07-31T21:01:59.998Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-07-31T21:16:32.721Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/marionette/reftest.js:327:23 | Replace ?lhs:?null,?rhs:?null,?passed:?false,?error:?e,?msg:?null with ???????????lhs:?null,???????????rhs:?null,???????????passed:?false,???????????error:?e,???????????msg:?null,???????? (prettier/prettier)
[task 2019-07-31T21:16:32.721Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/marionette/reftest.js:440:13 | Replace ??msg?= with msg?=?????????????? (prettier/prettier)
[task 2019-07-31T21:16:32.721Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/marionette/reftest.js:441:1 | Replace ????????????????maximum?difference?per?channel?${maxDifferences.value}`` with ??????????????maximum?difference?per?channel?${maxDifferences.value}`;` (prettier/prettier)
[taskcluster 2019-07-31 21:16:33.076Z] === Task Finished ===
[taskcluster 2019-07-31 21:16:33.733Z] Unsuccessful task run with exit code: 1 completed in 1133.08 seconds

Flags: needinfo?(james)
Flags: needinfo?(james)
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/ba33cb1e50f6
Report pixel differences as part of message in wpt reftests, r=maja_zf
Status: NEW → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
You need to log in before you can comment on or make changes to this bug.