1116972 - Intermittent Linux-32 reftest failures where data: URLs are string-equal but image comparison says they're not

Status: RESOLVED INCOMPLETE

(Testing :: Reftest, defect)

Description

[Phil Ringnalda (:philor)]

Attachment: reftest.log

660732 pixels seems like rather a lot to differ, particularly when you consider that the two images appear to be identical.

https://treeherder.mozilla.org/logviewer.html#?job_id=1573907&repo=fx-team

15:53:15 INFO - REFTEST TEST-START | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/svg/clipPath-and-transform-01.svg
15:53:15 INFO - REFTEST TEST-LOAD | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/svg/clipPath-and-transform-01.svg | 4047 / 5854 (69%)
15:53:15 INFO - ++DOMWINDOW == 55 (0x90235400) [pid = 3321] [serial = 11860] [outer = 0xa12bcc00]
15:53:16 INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/svg/clipPath-and-transform-01.svg | image comparison (==), max difference: 255, number of differing pixels: 660732
...
15:53:16 INFO - REFTEST INFO | Saved log: START file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/svg/clipPath-and-transform-01.svg
15:53:16 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering AfterOnLoadScripts
15:53:16 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
15:53:16 INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
15:53:16 INFO - REFTEST INFO | Saved log: [CONTENT] RecordResult fired
15:53:16 INFO - REFTEST INFO | Saved log: RecordResult fired
15:53:16 INFO - REFTEST INFO | Saved log: RecordResult fired
15:53:16 INFO - REFTEST INFO | Loading a blank page
15:53:16 INFO - ++DOMWINDOW == 56 (0x902e6200) [pid = 3321] [serial = 11861] [outer = 0xa12bcc00]
15:53:16 INFO - REFTEST TEST-END | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/svg/clipPath-and-transform-01.svg

Comment 1

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2015-01-01T08:44:35
log: https://treeherder.mozilla.org/ui/logviewer.html#?repo=fx-team&job_id=1573907
repository: fx-team
who: philringnalda[at]gmail[dot]com
machine: tst-linux32-spot-065
buildname: Ubuntu VM 12.04 fx-team debug test reftest-2
revision: c738f8368814

REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/svg/clipPath-and-transform-01.svg | image comparison (==), max difference: 255, number of differing pixels: 660732

Comment 2

[Legacy TBPL/Treeherder Robot]

submit_timestamp: 2015-01-17T15:39:20
log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=901855
repository: mozilla-central
who: philringnalda[at]gmail[dot]com
machine: tst-linux32-spot-229
buildname: Ubuntu VM 12.04 mozilla-central debug test reftest-2
revision: 41c672fbc3c1

REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/mathml/mathbackground-4.xml | image comparison (==), max difference: 255, number of differing pixels: 4

Comment 3

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

(In reply to TBPL Robot from comment #1)
> https://treeherder.mozilla.org/ui/logviewer.html#?repo=fx-team&job_id=1573907

The two data: URIs (for test and reference) string-compare as equal, which means there's something wrong with our diagnostics here...

Comment 4

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

And philor pointed out the same for the comment 2 logs prior to my comment 3.

Comment 5

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-inbound&job_id=11094412
repository: mozilla-inbound
start_time: 2015-06-24T18:16:42
who: philringnalda[at]gmail[dot]com
machine: tst-linux32-spot-520
buildname: Ubuntu VM 12.04 mozilla-inbound debug test reftest-e10s-1
revision: 3ef9ca698c52

REFTEST TEST-UNEXPECTED-FAIL | file:///builds/slave/test/build/tests/reftest/tests/layout/reftests/bugs/411585-2.html | image comparison (==), max difference: 146, number of differing pixels: 6

Comment 6

[Phil Ringnalda (:philor)]

Sort of a fun question, just how often we've filed intermittent failure bugs without looking at the lack of circled differences in the reftest analyzer, and then nobody has looked at the bug.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.