1445834 - Fix & re-enable css/CSS2/borders tests on Linux32 opt (skipped due to random toBase64@chrome://marionette/content/reftest.js:216:21 permafailing)

Status: REOPENED

(Core :: CSS Parsing and Computation, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=168077611&repo=mozilla-inbound

https://queue.taskcluster.net/v1/task/Q_ClGYFuQjqSpK8AjRf1UQ/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/Q_ClGYFuQjqSpK8AjRf1UQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2018-03-14T23:21:26.049Z] 23:21:26     INFO - TEST-START | /css/CSS2/borders/border-top-width-050.xht
[task 2018-03-14T23:21:26.057Z] 23:21:26     INFO - PID 2529 | 1521069686048	Marionette	INFO	Testing http://web-platform.test:8000/css/CSS2/borders/border-top-width-050.xht == http://web-platform.test:8000/css/CSS2/borders/border-bottom-width-006-ref.xht
[task 2018-03-14T23:21:26.078Z] 23:21:26     INFO - TEST-UNEXPECTED-ERROR | /css/CSS2/borders/border-top-width-050.xht | toBase64@chrome://marionette/content/reftest.js:216:21
[task 2018-03-14T23:21:26.078Z] 23:21:26     INFO - recordScreenshot@chrome://marionette/content/reftest.js:243:26
[task 2018-03-14T23:21:26.078Z] 23:21:26     INFO - runTest@chrome://marionette/content/reftest.js:275:11
[task 2018-03-14T23:21:26.078Z] 23:21:26     INFO - async*run/testRunner<@chrome://marionette/content/reftest.js:196:24
[task 2018-03-14T23:21:26.079Z] 23:21:26     INFO - async*run@chrome://marionette/content/reftest.js:193:23
[task 2018-03-14T23:21:26.079Z] 23:21:26     INFO - async*GeckoDriver.prototype.runReftest@chrome://marionette/content/driver.js:3478:27
[task 2018-03-14T23:21:26.079Z] 23:21:26     INFO - Async*despatch@chrome://marionette/content/server.js:288:20
[task 2018-03-14T23:21:26.079Z] 23:21:26     INFO - async*execute@chrome://marionette/content/server.js:262:11
[task 2018-03-14T23:21:26.080Z] 23:21:26     INFO - async*onPacket/<@chrome://marionette/content/server.js:237:15
[task 2018-03-14T23:21:26.080Z] 23:21:26     INFO - async*onPacket@chrome://marionette/content/server.js:236:8
[task 2018-03-14T23:21:26.080Z] 23:21:26     INFO - _onJSONObjectReady/<@chrome://marionette/content/transport.js:500:9
[task 2018-03-14T23:21:26.080Z] 23:21:26     INFO - TEST-INFO took 30ms
[task 2018-03-14T23:21:26.081Z] 23:21:26     INFO - PID 2529 | 1521069686072	Marionette	INFO	Found 360000 pixels different, maximum difference per channel 255
[task 2018-03-14T23:21:26.299Z] 23:21:26     INFO - PID 2529 | *** UTM:SVC TimerManager:registerTimer called after profile-before-change notification. Ignoring timer registration for id: telemetry_modules_ping
[task 2018-03-14T23:21:26.643Z] 23:21:26     INFO - Browser exited with return code 0
[task 2018-03-14T23:21:26.645Z] 23:21:26  WARNING - u'runner_teardown': ()
[task 2018-03-14T23:21:26.667Z] 23:21:26     INFO - Setting up ssl
[task 2018-03-14T23:21:26.695Z] 23:21:26     INFO - certutil | 
[task 2018-03-14T23:21:26.723Z] 23:21:26     INFO - certutil | 
[task 2018-03-14T23:21:26.743Z] 23:21:26     INFO - certutil | 
[task 2018-03-14T23:21:26.743Z] 23:21:26     INFO - Certificate Nickname                                         Trust Attributes
[task 2018-03-14T23:21:26.744Z] 23:21:26     INFO -                                                              SSL,S/MIME,JAR/XPI
[task 2018-03-14T23:21:26.744Z] 23:21:26     INFO - 
[task 2018-03-14T23:21:26.744Z] 23:21:26     INFO - web-platform-tests                                           CT,, 
[task 2018-03-14T23:21:26.744Z] 23:21:26     INFO - 
[task 2018-03-14T23:21:26.821Z] 23:21:26     INFO - Application command: /builds/worker/workspace/build/application/firefox/firefox --marionette about:blank -profile /tmp/tmptVlOAu.mozrunner
[task 2018-03-14T23:21:26.837Z] 23:21:26     INFO - Starting runner
[task 2018-03-14T23:21:28.600Z] 23:21:28     INFO - PID 3993 | 1521069688592	Marionette	INFO	Listening on port 2828

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Random developers have been hitting this lately. Smells like an OOM situation to me. Given that's a time-bomb waiting to hit random people and cause needless backouts, I'm skipping this directory on Linux32 opt for now.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/657ca6bb4eca
Skip css/CSS2/borders web-platform-tests on Linux32 opt due to frequent failures on a CLOSED TREE.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

I honestly have no idea if this is an issue with the harness or a Gecko issue. Which I know is going to make this very appealing for anybody to prioritize investigating. CCing a few people who might be interested, just in case.

Comment 4

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/657ca6bb4eca

Comment 5

[Intermittent Failures Robot]

11 failures in 814 pushes (0.014 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 11

Platform breakdown:
* linux32-stylo-disabled: 11

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1445834&startday=2018-03-12&endday=2018-03-18&tree=all

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The problem here happens in the `recordScreenshot()` method:

https://dxr.mozilla.org/mozilla-central/rev/6ff60a083701d08c52702daf50f28e8f46ae3a1c/testing/marionette/reftest.js#243

Specifically converting the `lhs` value to a base64 string.

From the log I see:

[task 2018-03-14T23:21:26.057Z] 23:21:26     INFO - PID 2529 | 1521069686048	Marionette	INFO	Testing http://web-platform.test:8000/css/CSS2/borders/border-top-width-050.xht == http://web-platform.test:8000/css/CSS2/borders/border-bottom-width-006-ref.xht

I don't know the data structure of `lhs` but something is going then wrong in the call to `screenshot.canvas.toDataURL();`. Sadly the real failure is hidden. Given that James added the reftest part to Marionette, he should know more than I do.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

This test is still disabled and the bug shouldn't have been closed.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I actually cannot see any disabled tests in that folder for Linux anymore:
https://searchfox.org/mozilla-central/source/testing/web-platform/meta/css/CSS2/borders

Emilio, looks like we can remove the test-disabled keyword and mark the bug as WFM?

Comment 10

[Emilio Cobos Álvarez (:emilio)]

Seems disabled to me: https://searchfox.org/mozilla-central/rev/5fdbe0d78d61581aa4b37ce82bbf31c2bd7826f1/testing/web-platform/meta/css/CSS2/borders/__dir__.ini#1-2

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Oh, I missed __dir__.ini!! You are right, but it looks like this is for Linux 32bit only and we seem to no longer run these jobs in CI. But --full still offers them. So we probably have to wait a bit longer until 32bit support has been completely removed.

Joel, do we have a tracking bug for that work?

Comment 12

[Joel Maher ( :jmaher ) (UTC -8)]

I don't know of any plans to remove 32 bit testing from CI. Probably something to look into next year sometime.