ubuntu1804 - high-contrast/backplate-bg-image-009.html == high-contrast/backplate-bg-image-009-ref.html | image comparison, max difference: 255, number of differing pixels: 1155
Categories
(Core :: Layout, defect, P5)
Tracking
()
People
(Reporter: intermittent-bug-filer, Unassigned)
References
Details
Attachments
(1 file)
Filed by: egao [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=271937747&repo=try
Full log: https://queue.taskcluster.net/v1/task/ZzSzlxrYRf6IPj2LVkHEXA/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/ZzSzlxrYRf6IPj2LVkHEXA/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
platform: debian 10
suite: reftest-2
context:
[task 2019-10-18T17:23:57.455Z] 17:23:57 INFO - REFTEST TEST-START | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/high-contrast/backplate-bg-image-009.html == file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/high-contrast/backplate-bg-image-009-ref.html
[task 2019-10-18T17:23:57.455Z] 17:23:57 INFO - REFTEST INFO | SET PREFERENCE pref(browser.display.document_color_use,2)
[task 2019-10-18T17:23:57.455Z] 17:23:57 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/high-contrast/backplate-bg-image-009.html | 8 / 9 (88%)
[task 2019-10-18T17:23:57.455Z] 17:23:57 INFO - REFTEST INFO | RESTORE PREFERENCE pref(browser.display.document_color_use,0)
[task 2019-10-18T17:23:57.455Z] 17:23:57 INFO - REFTEST TEST-LOAD | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/high-contrast/backplate-bg-image-009-ref.html | 8 / 9 (88%)
[task 2019-10-18T17:23:57.487Z] 17:23:57 INFO - REFTEST INFO | REFTEST fuzzy test (255, 1050) <= (255, 1291) <= (255, 1050)
[task 2019-10-18T17:23:57.626Z] 17:23:57 INFO - REFTEST TEST-UNEXPECTED-FAIL | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/high-contrast/backplate-bg-image-009.html == file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/high-contrast/backplate-bg-image-009-ref.html | image comparison, max difference: 255, number of differing pixels: 1291
|
||
Comment 1•5 years ago
|
||
Updating the bug title to reflect the fact that debian10 work has been stopped and focus is now on ubuntu1804, and on that platform the pixel variation values are slightly different.
:morgan - on searchfox it appears the test was recently added. It appears that on ubuntu1804 (which is what I am working to migrate our linux CI to) the pixel variations produced are slightly higher than that of ubuntu1604.
Would you prefer that I update the test manifest once I make the switchover to have linux tests run on ubuntu1804, or take a look at the cause of the discrepancy? Ideally, I would have a condition for ubuntu1804 in the manifest (like I discussed with dholbert a few months ago) but I have not been able to implement something like that.
|
||
Comment 2•5 years ago
|
||
Note that the Try logs in comment 0 have expired and can't be used to visualize the failure anymore.
Here's a Try run from yesterday that works for now:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=278325920&repo=try&lineNumber=14377
And I'm attaching the relevant snippet (with the unexpected results for the two backplate tests here), so that we've got that handy even after the Try log expires. (The snippet can just be pasted directly into https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml for processing.)
|
|
||
Comment 3•5 years ago
|
||
In both cases here, the tests are annotated as pretty fuzzy, and we're just outside of the fuzzy range. We're just under the fuzzy range, in the -008 test (only 600px mismatching, which is less than the allowed 620px), and we're just over the fuzzy range in the -009 test [1155px mismatching, which is more than the allowed 1050px]
Here's the snippet of the log that I pulled those numbers from (with some bits snipped out for brevity):
REFTEST fuzzy test (255, 620) <= (255, 600) <= (255, 620)
REFTEST TEST-UNEXPECTED-PASS | file:///.../backplate-bg-image-008.html
[...]
REFTEST fuzzy test (255, 1050) <= (255, 1155) <= (255, 1050)
REFTEST TEST-UNEXPECTED-FAIL | file:///.../backplate-bg-image-009.html ...
This is likely explained by the difference in GTK themes between our current linux testrunners vs. the new Ubuntu 18.04 test runners.
The appropriate fix for that would just be a change in the allowable fuzzy range, but we shouldn't do that at this point because Morgan is changing these tests in https://phabricator.services.mozilla.com/D52065 to supposedly stop needing a fuzzy annotation at all on Linux.
Edwin, would you mind importing that ^ patch from phabricator, and giving this a run through Try to see if you still see unexpected results in these backplate tests?
|
|
||
Comment 4•5 years ago
•
|
||
I have pushed a new run to try with the patch imported and running on ubuntu1804:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7d985147fbdcbbd900ac0af8388267531ee6e245
Edit: Looks like reftest-e10s-4 is the one where the backplate-bg-image test are run in this push.
|
|
||
Comment 5•5 years ago
•
|
||
Thanks. Sample run with those backplate test failures (for as long as the Try run's logs are around):
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=278503835&repo=try&lineNumber=20893
Looks like we'll need to add back some annotations for (and/or tweaks to) lots of the backplate tests, after bug 1594099 lands, in order for them to be green on this Ubuntu 18.04 platform. (Though this doesn't need to block the changes in bug 1594099, because these 18.04 test runners aren't official yet.)
|
||
Comment 6•4 years ago
|
||
hello! I'm working on annotating the (slightly-modified) tests for 1594099 right now, I'll make sure this gets annotated in that sweep. Current try to gauge fuzziness here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=c7a10d9c90a00e5e561ba35a56fc7dba640f507e
|
|
||
Comment 7•4 years ago
|
||
Thanks!
The ubuntu1804 image is in-tree, so if you want to ever push against ubuntu1804 to test, please use the following:
./mach try fuzzy --ubuntu-bionic
It will toggle the image used from desktop1604-test (default) to ubuntu1804-test.
|
|
||
Comment 8•4 years ago
|
||
(In reply to Edwin Takahashi (:egao, :etakahashi) from comment #7)
Thanks!
The ubuntu1804 image is in-tree, so if you want to ever push against ubuntu1804 to test, please use the following:
./mach try fuzzy --ubuntu-bionicIt will toggle the image used from
desktop1604-test(default) toubuntu1804-test.
After running that, should I select the -linux labelled things as normal? Or is there something else to do in the prompt?
|
|
||
Updated•4 years ago
|
|
|
||
Comment 9•4 years ago
|
||
:morgan - yes, select the linux jobs as normal. Keep in mind that if you want results for both 1804 and 1604 you'd have to push to try twice.
|
|
||
Comment 10•4 years ago
|
||
In-progress try with changes from 1594099 for ubuntu1804:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=f61c3c2144556a3d15cc5bd8678f99ce4c6f2a70&selectedJob=279428832
Looks like I need to change the range for 1604 down to 15-24 from 24-24, based on this:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a4478487a29b31d00cc645904110eb62d69f8632&selectedJob=279427505
but will do :)
|
|
||
Comment 11•4 years ago
•
|
||
With the dependent bug 1594099 fixed, I pushed to try with the fix:
https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedJob=280820896&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cretry%2Crunning%2Cpending%2Crunnable&tier=1%2C2%2C3&revision=0afef5b0a00c4aa5595cbf2d456aba9fc6d7edb1
It looks like the backplate related issues are no longer an issue on ubuntu1804 image!
I will be closing this bug unless the issue resurfaces.
Description
•