1134966 - Intermittent 1133392.html | image comparison (==), max difference: 255, number of differing pixels: 93

Status: RESOLVED DUPLICATE of bug 1123025

(Core :: Layout, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Attachment: test screenshot

20:29:12 INFO - REFTEST TEST-START | http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html
20:29:12 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html | 524 / 549 (95%)
20:29:12 INFO - REFTEST TEST-LOAD | http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392-ref.html | 524 / 549 (95%)
20:29:12 INFO - REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html | image comparison (==), max difference: 255, number of differing pixels: 93
20:29:12 INFO - REFTEST IMAGE 1 (TEST): 
20:29:13 INFO - REFTEST IMAGE 2 (REFERENCE): 
20:29:13 INFO - REFTEST INFO | Saved log: START http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering WaitForTestEnd
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] WaitForTestEnd: Adding listeners
20:29:13 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
20:29:13 INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for MozAfterPaint
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] AfterPaintListener in http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] SendUpdateCanvasForEvent with 1 rects
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] Rect: 0 0 800 1000
20:29:13 INFO - REFTEST INFO | Saved log: Updating canvas for invalidation
20:29:13 INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FIRE_INVALIDATE_EVENT
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: dispatching MozReftestInvalidate
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] AttrModifiedListener fired
20:29:13 INFO - REFTEST INFO | Saved log: [CONTENT] AttrModifiedListener fired
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] FlushRendering generated paint for window http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_REFTEST_WAIT_REMOVAL
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_FOR_SPELL_CHECKS
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FINISH
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for MozAfterPaint
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FINISH
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for MozAfterPaint
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FINISH
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: waiting for MozAfterPaint
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] AfterPaintListener in http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] SendUpdateCanvasForEvent with 1 rects
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] Rect: 36 66 131 87
20:29:33 INFO - REFTEST INFO | Saved log: Updating canvas for invalidation
20:29:33 INFO - REFTEST INFO | Saved log: DoDrawWindow 36,66,95,21
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: STATE_WAITING_TO_FINISH
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] MakeProgress: Completed
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] RecordResult fired
20:29:33 INFO - REFTEST INFO | Saved log: RecordResult fired
20:29:33 INFO - REFTEST INFO | Saved log: START http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392-ref.html
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] OnDocumentLoad triggering AfterOnLoadScripts
20:29:33 INFO - REFTEST INFO | Saved log: Initializing canvas snapshot
20:29:33 INFO - REFTEST INFO | Saved log: DoDrawWindow 0,0,800,1000
20:29:33 INFO - REFTEST INFO | Saved log: [CONTENT] RecordResult fired
20:29:33 INFO - REFTEST INFO | Saved log: RecordResult fired
20:29:33 INFO - REFTEST INFO | Loading a blank page
20:29:33 INFO - REFTEST TEST-END | http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Attachment: reference screenshot

Comment 2

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1059002
repository: mozilla-central
start_time: 2015-02-19T19:28:47
who: rvandermeulen[at]mozilla[dot]com
machine: tst-linux64-spot-627
buildname: b2g_emulator_vm mozilla-central opt test reftest-19
revision: 51458a066fda

REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html | image comparison (==), max difference: 255, number of differing pixels: 93
Return code: 1

Comment 3

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1059012
repository: mozilla-central
start_time: 2015-02-19T20:46:58
who: rvandermeulen[at]mozilla[dot]com
machine: tst-linux64-spot-668
buildname: b2g_emulator_vm mozilla-central opt test reftest-19
revision: 51458a066fda

REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html | image comparison (==), max difference: 255, number of differing pixels: 93
Return code: 1

Comment 4

[Legacy TBPL/Treeherder Robot]

log: https://treeherder.mozilla.org/logviewer.html#?repo=mozilla-central&job_id=1059013
repository: mozilla-central
start_time: 2015-02-19T20:46:58
who: rvandermeulen[at]mozilla[dot]com
machine: tst-linux64-spot-628
buildname: b2g_emulator_vm mozilla-central opt test reftest-19
revision: 51458a066fda

REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/text-decoration/1133392.html | image comparison (==), max difference: 255, number of differing pixels: 93
Return code: 1

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Interesting that it went away after two days.

Are we really sure that our B2G emulator tests are testing the Gecko that they claim to be testing?  Because the failure looks exactly like what would happen if you ran the test in a build that didn't have bug 1133392's fix in it.

Comment 6

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

from log in comment 2:
19:29:00     INFO - Downloading http://pvtbuilds.mozilla.org.proxxy1.srv.releng.usw2.mozilla.com/pub/mozilla.org/b2g/nightly/mozilla-central-emulator/latest/b2g-38.0a1.en-US.android-arm.tests.zip to /builds/slave/test/build/b2g-38.0a1.en-US.android-arm.tests.zip
19:29:14     INFO - Downloading http://pvtbuilds.mozilla.org.proxxy1.srv.releng.usw2.mozilla.com/pub/mozilla.org/b2g/nightly/mozilla-central-emulator/latest/emulator.tar.gz to /builds/slave/test/build/emulator.tar.gz
from log in comment 3:
20:47:23     INFO - Downloading http://pvtbuilds.mozilla.org.proxxy1.srv.releng.usw2.mozilla.com/pub/mozilla.org/b2g/nightly/mozilla-central-emulator/latest/b2g-38.0a1.en-US.android-arm.tests.zip to /builds/slave/test/build/b2g-38.0a1.en-US.android-arm.tests.zip
20:47:37     INFO - Downloading http://pvtbuilds.mozilla.org.proxxy1.srv.releng.usw2.mozilla.com/pub/mozilla.org/b2g/nightly/mozilla-central-emulator/latest/emulator.tar.gz to /builds/slave/test/build/emulator.tar.gz
from log in comment 4:
20:47:16     INFO - Downloading http://pvtbuilds.mozilla.org.proxxy1.srv.releng.usw2.mozilla.com/pub/mozilla.org/b2g/nightly/mozilla-central-emulator/latest/b2g-38.0a1.en-US.android-arm.tests.zip to /builds/slave/test/build/b2g-38.0a1.en-US.android-arm.tests.zip
20:47:32     INFO - Downloading http://pvtbuilds.mozilla.org.proxxy1.srv.releng.usw2.mozilla.com/pub/mozilla.org/b2g/nightly/mozilla-central-emulator/latest/emulator.tar.gz to /builds/slave/test/build/emulator.tar.gz

I am rather suspicious of the "latest" in these URLs.

philor points out 1123025.

And also (from #developers):

[2015-03-02 15:47:08 -0800] <dbaron> RyanVM, how confident are we that our B2G emulator reftest runs are testing matching builds and reftests?
[2015-03-02 15:47:14 -0800] <RyanVM> dbaron: heh
[2015-03-02 15:47:24 -0800] <RyanVM> philor: would you like to answer that? :D
[2015-03-02 15:49:23 -0800] <philor> dbaron:  well, let's see: for "nightly" builds, we're nearly certain that they do not correctly match up; for any retriggered build, it's pure coincidence and luck of the draw if they match up
[2015-03-02 15:49:42 -0800] <dbaron> philor, is there a bug on this that I can dupe something to?
[2015-03-02 15:50:01 -0800] * philor considers what rage phrasing he would have used while filing it
[2015-03-02 15:50:42 -0800] <dbaron> (though I'm still going through the log to double-check that that's actually what happened... but when an intermittent reftest failure gets filed that fails just like the patch wasn't there, only on b2g emulator, only a few times, and only in the day and a half after the fix lands...)
[2015-03-02 15:51:52 -0800] <philor> dbaron:  the nightly thing, https://bugzilla.mozilla.org/show_bug.cgi?id=1123025
[2015-03-02 15:52:17 -0800] <dbaron> philor, what I saw would be newer tests and older build
[2015-03-02 15:52:42 -0800] <philor> hmm
[2015-03-02 15:53:07 -0800] <dbaron> philor, it's https://bugzilla.mozilla.org/show_bug.cgi?id=1134966, btw
[2015-03-02 15:54:23 -0800] <dbaron> I'm quite concerned from the bit in the log that has it downloading the build and tests from a directory called "latest"
[2015-03-02 15:54:32 -0800] <philor> dbaron: that should be possible if someone triggered builds on an older rev, since yeah, latest
[2015-03-02 15:57:27 -0800] <philor> odd, https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=51458a066fda only has one set of tests, usually looking at this should start with the problem of telling which test came from the dep build and which from the nightly 
[2015-03-02 15:58:06 -0800] <philor> oh, hello coalescing
[2015-03-02 15:58:31 -0800] <philor> the on-push build started at 15:57, the "nightly" at 16:02
[2015-03-02 15:58:41 -0800] <philor> ... you aren't going to have a good time

Comment 7

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

I also looked at an R19 log for a different push, and it wasn't attempting to download the emulator and test packages from "latest".