1439115 - http://10.0.2.2:8888/tests/layout/reftests/css-disabled/select/select-fieldset-4.html == http://10.0.2.2:8888/tests/layout/reftests/css-disabled/select/select-fieldset-ref.html | image comparison, max difference: 12, number of differing pixels: 1

Status: RESOLVED FIXED

(Core :: Layout: Form Controls, defect, P3)

Description

[Geoff Brown [:gbrown] (pto Apr 11-18)]

As seen in https://bugzilla.mozilla.org/show_bug.cgi?id=1433163#c26, 2 css-disabled/select reftests become near perma-fail on Android when test tasks are modified to change the way tests are divided amongst tasks. It looks like these tests depend on state set up by earlier tests.


Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b2fe34d6f8a054fc0f4f64740ad4f984cf647510

Failure log for R13 test: https://treeherder.mozilla.org/logviewer.html#?job_id=162726354&repo=mozilla-inbound&lineNumber=1557
[task 2018-02-16T20:25:45.365Z] 20:25:45     INFO -  REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/css-disabled/select/select-fieldset-4.html == http://10.0.2.2:8888/tests/layout/reftests/css-disabled/select/select-fieldset-ref.html | image comparison, max difference: 12, number of differing pixels: 1

Failure log for R25 test: https://treeherder.mozilla.org/logviewer.html#?job_id=162726661&repo=mozilla-inbound&lineNumber=1571
[task 2018-02-16T20:33:49.469Z] 20:33:49     INFO -  REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8888/tests/layout/reftests/css-disabled/select/select-fieldset-2.html == http://10.0.2.2:8888/tests/layout/reftests/css-disabled/select/select-fieldset-ref-disabled.html | image comparison, max difference: 12, number of differing pixels: 1

Comment 1

[Geoff Brown [:gbrown] (pto Apr 11-18)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=93c46cd871a0ae4cc0f09ebf072b985ca98379f2 demonstrates that these tests fail test-verify: They cannot be run reliably on their own.

Comment 2

[Jet Villegas (inactive)]

[ Triage 2017/02/20: P3 ]

Comment 3

[Geoff Brown [:gbrown] (pto Apr 11-18)]

There was a similar issue hit in bug 1295261, resolved by fuzzy-if(Android). I'll do the same here.

Passes test-verify with the fuzzy-if:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=008fd8950cf6cd862af90eacc7fdb3bea987e4d7

and unblocks bug 1433163:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=3f5a5cdf97f1a5996c09775f6e73e080f23ac206

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/42248666ddf6
Annotate css-disabled/select reftests so they pass reliably on Android; r=me,a=test-only

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/671b0faa7112
(follow-up) Annotate more css-disabled/select reftests so they pass reliably on Android; r=me,a=test-only

Comment 6

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/42248666ddf6
https://hg.mozilla.org/mozilla-central/rev/671b0faa7112