1425382 - failure to generate message with difference count for pixels

Status: NEW

(Remote Protocol :: Marionette, defect, P5)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

See https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=pending&filter-resultStatus=running&tochange=6cb9414b5d77ff1bc333274b554d7296ab3d5f09&fromchange=d3917d82652d8559ade46358f197cfe69b78afdf&filter-searchStr=wr4

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=151720058&repo=autoland

[task 2017-12-14T23:40:50.755Z] 23:40:50     INFO - TEST-START | /css/CSS2/borders/border-top-color-136.xht
[task 2017-12-14T23:40:50.763Z] 23:40:50     INFO - PID 2589 | 1513294850757	Marionette	INFO	Testing http://web-platform.test:8000/css/CSS2/borders/border-top-color-136.xht == http://web-platform.test:8000/css/CSS2/borders/border-bottom-color-136-ref.xht
[task 2017-12-14T23:40:50.787Z] 23:40:50     INFO - PID 2589 | 1513294850783	Marionette	INFO	Found 360000 pixels different, maximum difference per channel 255
[task 2017-12-14T23:40:50.788Z] 23:40:50     INFO - TEST-UNEXPECTED-ERROR | /css/CSS2/borders/border-top-color-136.xht | toBase64@chrome://marionette/content/reftest.js:218:21
[task 2017-12-14T23:40:50.788Z] 23:40:50     INFO - recordScreenshot@chrome://marionette/content/reftest.js:245:26
[task 2017-12-14T23:40:50.789Z] 23:40:50     INFO - runTest@chrome://marionette/content/reftest.js:277:11
[task 2017-12-14T23:40:50.789Z] 23:40:50     INFO - async*run/testRunner<@chrome://marionette/content/reftest.js:198:24
[task 2017-12-14T23:40:50.789Z] 23:40:50     INFO - async*run@chrome://marionette/content/reftest.js:195:23
[task 2017-12-14T23:40:50.789Z] 23:40:50     INFO - async*GeckoDriver.prototype.runReftest@chrome://marionette/content/driver.js:3468:27
[task 2017-12-14T23:40:50.789Z] 23:40:50     INFO - Async*despatch@chrome://marionette/content/server.js:557:20
[task 2017-12-14T23:40:50.789Z] 23:40:50     INFO - async*execute@chrome://marionette/content/server.js:531:11
[task 2017-12-14T23:40:50.790Z] 23:40:50     INFO - async*onPacket/<@chrome://marionette/content/server.js:506:15
[task 2017-12-14T23:40:50.790Z] 23:40:50     INFO - async*onPacket@chrome://marionette/content/server.js:505:8
[task 2017-12-14T23:40:50.790Z] 23:40:50     INFO - _onJSONObjectReady/<@chrome://marionette/content/transport.js:500:9
[task 2017-12-14T23:40:50.791Z] 23:40:50     INFO - TEST-INFO took 31ms

There used to be messages about the number of different pixels.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

James can you have a look please?

Comment 2

[James Graham [:jgraham]]

The test failed, then we tried to generate a data url of the canvas for the logs, and the conversion to a png for some reason. It would be nice if marionette would record the actual exception as well as the stack trace. The relevant code hasn't changed recently so I expect this is a very infrequent intermittent that could be e.g. OOM. But if I'm wrong and it happens more often than I should look at it again.

Comment 3

[Andreas Tolfsen ❲:ato❳]

(In reply to James Graham [:jgraham] from comment #2)

> It would be nice if marionette would record the actual exception
> as well as the stack trace.

We do record the error message, but not sure why it’s not turning up
in the harness logs.  I suspect this is a harness problem.

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to James Graham [:jgraham] from comment #2)
> The
> relevant code hasn't changed recently so I expect this is a very infrequent
> intermittent that could be e.g. OOM. But if I'm wrong and it happens more
> often than I should look at it again.
This showed up in masses, see the treeherder link in comment 0.

Comment 5

[David Burns :automatedtester]

[Mass Change 2018-01-15] Moving bugs to backlog

Comment 6

[Andreas Tolfsen ❲:ato❳]

I wonder if something could have broken temporarily in Firefox,
becuase as far as I can tell this is not an intermittent?