1563358 - Android web-platform-tests-reftests logs don't report maxDifference or totalPixels differing, alongside reftest failures

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect, P2)

Description

[Daniel Holbert [:dholbert]]

In e.g. this Try run with wpt reftest failures...
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ec7224ebb3e39351e49a900442f99804750238e

...the test failures look like this on most platforms:

INFO - TEST-START | /css/css-contain/contain-paint-clip-015.html
[...]
INFO - PID 8580 | 1562178423597	Marionette	INFO	No differences allowed
INFO - PID 8580 | 1562178423598	Marionette	INFO	Found 332 pixels different, maximum difference per channel 98
INFO - TEST-UNEXPECTED-FAIL | /css/css-contain/contain-paint-clip-015.html | [...]

But on android, that "No differences allowed" / "Found 332 pixels different, maximum difference per channel 98" text is entirely missing. On that platform, there's nothing between the TEST-START and TEST-UNEXPECTED-FAIL lines. This makes it hard to know the right values to use for fuzzy annotations on that platform.

If possible, it'd be great to get the "$X differences allowed" and "$Y pixels-different $Z max-difference" values reported in that log.

Comment 1

[Daniel Holbert [:dholbert]]

(It looks like this logging is present in the "logcat" artifact for these runs, so it's there -- it's just a bit buried.)

Comment 2

[BugBot [:suhaib / :marco]]

The priority flag is not set for this bug.
:jgraham, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 3

[James Graham [:jgraham]]

So basically it seems like the marionette logs end up in the logcat (which is a different artifact) rather than the test output. Which means this is available but is inconvenient to find.

majazf: Can you look at this? I think one solution would be to add this information to the failure message so that it's sent back to the harness and printed there rather than being printed in the marionette logging.

Comment 4

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

We agreed that James will fix this as part of other Android wpt stuff he's going to work on soon.

Comment 5

[James Graham [:jgraham]]

Attachment: Bug 1563358 - Report pixel differences as part of message in wpt reftests,

Instead of reporting the pixel differences in a logger info message,
put it in the text that's returned to the harness. This has a notable
advantage on android where this will cause it to be logged as part of
the harness logs rather than ending up in the logcat from the
device. It also makes these messages more accessible in other
consumers of the logs e.g. wpt.fyi.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f5f5cbb377ac
Report pixel differences as part of message in wpt reftests, r=maja_zf

Comment 7

[Raul Gurzau (:RaulG)]

Backed out changeset f5f5cbb377ac (bug 1563358) for eslint failure at /marionette/reftest.js on a CLOSED TREE.

Backout link: https://hg.mozilla.org/integration/autoland/rev/2e724ef1c10286e537104d6d812480611e680c81

**Push with failures:**https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=f5f5cbb377ac92a59c26a0f1de8b971ef878dee3&selectedJob=259261764

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=259261764&repo=autoland&lineNumber=234

Log snippet:

[task 2019-07-31T21:01:59.998Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-07-31T21:01:59.998Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-07-31T21:01:59.998Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-07-31T21:01:59.998Z]
[task 2019-07-31T21:01:59.998Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-07-31T21:16:32.721Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/marionette/reftest.js:327:23 | Replace ?lhs:?null,?rhs:?null,?passed:?false,?error:?e,?msg:?null with ???????????lhs:?null,???????????rhs:?null,???????????passed:?false,???????????error:?e,???????????msg:?null,???????? (prettier/prettier)
[task 2019-07-31T21:16:32.721Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/marionette/reftest.js:440:13 | Replace ??msg?= with msg?=?????????????? (prettier/prettier)
[task 2019-07-31T21:16:32.721Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/testing/marionette/reftest.js:441:1 | Replace ????????????????maximum?difference?per?channel?${maxDifferences.value}`` with ??????????????maximum?difference?per?channel?${maxDifferences.value}`;` (prettier/prettier)
[taskcluster 2019-07-31 21:16:33.076Z] === Task Finished ===
[taskcluster 2019-07-31 21:16:33.733Z] Unsuccessful task run with exit code: 1 completed in 1133.08 seconds

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ba33cb1e50f6
Report pixel differences as part of message in wpt reftests, r=maja_zf

Comment 9

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/ba33cb1e50f6