1604583 - ubuntu1804 - layout/reftests/position-sticky/reftest.list | multiple failures, adjust expectations?

Status: RESOLVED FIXED

(Core :: Layout: Positioned, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: egao [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=281172953&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fZMxbBhoS2eKG9aNjEFffA/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/fZMxbBhoS2eKG9aNjEFffA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

suite: reftest
chunk: 5

notes: qr

context:
[task 2019-12-14T09:54:37.350Z] 09:54:37 INFO - REFTEST TEST-START | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/position-sticky/transformed-2.html == file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/position-sticky/transformed-2-ref.html
[task 2019-12-14T09:54:37.351Z] 09:54:37 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/position-sticky/transformed-2.html | 52 / 55 (94%)
[task 2019-12-14T09:54:37.479Z] 09:54:37 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/position-sticky/transformed-2-ref.html | 52 / 55 (94%)
[task 2019-12-14T09:54:37.702Z] 09:54:37 INFO - REFTEST INFO | REFTEST fuzzy test (31, 24) <= (17, 32) <= (31, 24)
[task 2019-12-14T09:54:37.920Z] 09:54:37 INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/position-sticky/transformed-2.html == file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/position-sticky/transformed-2-ref.html | image comparison, max difference: 17, number of differing pixels: 32

Comment 1

[Edwin Takahashi (:egao | infrequent contributor)]

:svoisen - please feel free to redirect my ni elsewhere if appropriate.

Multiple tests from the refetst manifest layout/reftests/position-sticky/reftest.list are showing small deviations from the tight annotations in the manifest.

Looking at the reftest analyzer, most of these failures in the reftest-5 chunk are due to scrollbar difference.

My proposal is to expand the annotation range if required while ubuntu1804 and ubuntu1604 both exist in the CI system; once ubuntu1604 is decommissioned, then I will return and tighten up the annotation.

With the expected migration date of end of December 2019, I am hoping to have this marked and annotated as soon as possible.

To push to try with ubuntu1804, use ./mach try fuzzy --ubuntu-bionic and select test-linux64 jobs as normal.

Comment 2

[Sean Voisen (:svoisen)]

:egao — Sounds like a good plan to tighten up again after ubuntu1604 is decommissioned. Please go ahead with that here and where you run into similar issues with reftests.

Comment 4

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1604583 - add fuzzier annotation for layout/reftests/position-sticky/reftest.list on ubuntu1804

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a900b7526c68
add fuzzier annotation for layout/reftests/position-sticky/reftest.list on ubuntu1804 r=jmaher

Comment 6

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/a900b7526c68

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

== Change summary for alert #24500 (as of Mon, 16 Dec 2019 13:09:47 GMT) ==

Improvements:

8% perf_reftest some-descendants-1.html macosx1014-64-shippable opt e10s stylo 8.87 -> 8.20

For up to date results, see: https://treeherder.mozilla.org/perf.html#/alerts?id=24500