Closed Bug 1599943 Opened 5 years ago Closed 4 years ago

ubuntu1804 - widget/tests/test_scrollbar_colors.html | Pixel count of color 255,255,0 mismatches

Categories

(Core :: Widget, defect, P5)

Product:
Core
Component:
Widget
Type:
defect

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: intermittent-bug-filer, Assigned: egao)

References

Details

Attachments

(2 files)

Bug 1599943 - mark widget/tests/test_scrollbar_colors.html as expected failure on ubuntu1804
47 bytes, text/x-phabricator-request
Details | Review
Bug 1599943 - update pixel count expectations and re-enable test_scrollbar_colors.html for linux1804
47 bytes, text/x-phabricator-request
Details | Review

Filed by: egao [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=278537917&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/chcHudeXRnesRqfj5m-VMw/runs/0/artifacts/public/logs/live_backing.log

suite: mochitest
chunk: 5

context:

this test is now failing on ubunut1804 image.

[task 2019-11-27T23:57:40.667Z] 23:57:40 INFO - TEST-START | widget/tests/test_scrollbar_colors.html
[task 2019-11-27T23:57:40.688Z] 23:57:40 INFO - TEST-INFO | started process screentopng
[task 2019-11-27T23:57:41.073Z] 23:57:41 INFO - TEST-INFO | screentopng: exit 0
[task 2019-11-27T23:57:41.074Z] 23:57:41 INFO - TEST-UNEXPECTED-FAIL | widget/tests/test_scrollbar_colors.html | Pixel count of color 255,255,0 - got 7744, expected 8100
[task 2019-11-27T23:57:41.075Z] 23:57:41 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:322:16
[task 2019-11-27T23:57:41.076Z] 23:57:41 INFO - @widget/tests/test_scrollbar_colors.html:95:9
[task 2019-11-27T23:57:41.077Z] 23:57:41 INFO - Async*@widget/tests/test_scrollbar_colors.html:74:17
[task 2019-11-27T23:57:41.078Z] 23:57:41 INFO - Not taking screenshot here: see the one that was previously logged
[task 2019-11-27T23:57:41.078Z] 23:57:41 INFO - TEST-UNEXPECTED-FAIL | widget/tests/test_scrollbar_colors.html | Pixel count of color 0,0,255 - got 1104, expected 720
[task 2019-11-27T23:57:41.079Z] 23:57:41 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:322:16
[task 2019-11-27T23:57:41.080Z] 23:57:41 INFO - @widget/tests/test_scrollbar_colors.html:95:9
[task 2019-11-27T23:57:41.081Z] 23:57:41 INFO - Async*@widget/tests/test_scrollbar_colors.html:74:17
[task 2019-11-27T23:57:41.082Z] 23:57:41 INFO - Not taking screenshot here: see the one that was previously logged
[task 2019-11-27T23:57:41.082Z] 23:57:41 INFO - TEST-UNEXPECTED-FAIL | widget/tests/test_scrollbar_colors.html | Pixel count of color 0,255,255 - got 1152, expected 1180
[task 2019-11-27T23:57:41.083Z] 23:57:41 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:322:16
[task 2019-11-27T23:57:41.084Z] 23:57:41 INFO - @widget/tests/test_scrollbar_colors.html:95:9
[task 2019-11-27T23:57:41.085Z] 23:57:41 INFO - Async*@widget/tests/test_scrollbar_colors.html:74:17
[task 2019-11-27T23:57:41.086Z] 23:57:41 INFO - GECKO(10475) | MEMORY STAT | vsize 2584MB | residentFast 168MB | heapAllocated 34MB
[task 2019-11-27T23:57:41.086Z] 23:57:41 INFO - TEST-OK | widget/tests/test_scrollbar_colors.html | took 82ms

:xidorn - I am observing this test failing in a consistent manner on ubuntu1804 test image in the non-headless mode.

I noticed that previously, the test manifest file had the test skipped outright on linux platforms. Would it make sense to go back to that, from the current specification of os == 'linux' && headless?

Flags: needinfo?(xidorn+moz)
Summary: ubuntu1804 - widget/tests/test_scrollbar_colors.html → ubuntu1804 - widget/tests/test_scrollbar_colors.html | Pixel count of color 255,255,0 mismatches

This failure remains consistently reproducible.

try: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedJob=280054823&revision=5e7288f8c901d2fcf3eb35778f9d93e20aa5a151
log: https://firefoxci.taskcluster-artifacts.net/dFFma52MQ1mKcuhglfszjg/0/public/logs/live_backing.log

Current migration timeline to ubuntu1804 is at end of December 2019. This test will have the expectation updated or disabled closer to the date.

This test is just checking some number against a known values to catch unexpected regression.

In this case, the numbers change because of different OS, which is kinda expected, so the only thing we need to do is to update the number.

Is there any way to distinguish between the current Linux environment and the new Ubuntu 18.04 environment in mochitest? If so, we can just add a new condition in it with the new values. Otherwise, maybe you can disable this on Linux for now, and we can enable it again with the new values once the migration finishes.

Flags: needinfo?(egao)

Sounds good. I will mark the test to be skipped for only ubunti1804, then make a reminder to return to this bug and enter the updated color values once the switchover to ubuntu1804 is complete.

Thanks!

Flags: needinfo?(xidorn+moz)
Flags: needinfo?(egao)
Assignee: nobody → egao

The patch to mark the test as an expected failure is landing.

I will ni myself to mark that followup action needs to be taken to update the expected values for this test for ubuntu1804.

Flags: needinfo?(egao)
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9afbbd53c751
mark widget/tests/test_scrollbar_colors.html as expected failure on ubuntu1804 r=jmaher
Attachment #9132994 - Attachment description: Bug 1599943 - update pixel count expectations of test_scrollbar_colors.html for linux1804 → Bug 1599943 - update pixel count expectations and re-enable test_scrollbar_colors.html for linux1804
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/309c896d7f8e
update pixel count expectations and re-enable test_scrollbar_colors.html for linux1804 r=intermittent-reviewers,jmaher

Backed out changeset 309c896d7f8e (Bug 1599943) for linux tsan mochitest failures on test_scrollbar_colors.html.

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedJob=292965242&resultStatus=testfailed%2Cbusted%2Cexception&revision=309c896d7f8e4a433462b39c4cf7397473090cdb&searchStr=linux64-tsan%2Copt%2Cmochitests%2Ctest-linux64-tsan%2Fopt-mochitest-plain-e10s-12%2Cm%2812%29

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=292965242&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/45bdbb2f0d7c5846d4d360bb345faac394539781

[task 2020-03-13T02:00:17.669Z] 02:00:17 INFO - TEST-OK | widget/tests/test_assign_event_data.html | took 7066ms
[task 2020-03-13T02:00:17.853Z] 02:00:17 INFO - TEST-START | widget/tests/test_scrollbar_colors.html
[task 2020-03-13T02:00:18.428Z] 02:00:18 INFO - TEST-INFO | started process screentopng
[task 2020-03-13T02:00:19.059Z] 02:00:19 INFO - TEST-INFO | screentopng: exit 0
[task 2020-03-13T02:00:19.060Z] 02:00:19 INFO - TEST-UNEXPECTED-FAIL | widget/tests/test_scrollbar_colors.html | Pixel count of color 255,255,0 - got 8100, expected 7744
[task 2020-03-13T02:00:19.060Z] 02:00:19 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:383:14
[task 2020-03-13T02:00:19.060Z] 02:00:19 INFO - @widget/tests/test_scrollbar_colors.html:96:9
[task 2020-03-13T02:00:19.060Z] 02:00:19 INFO - Async*@widget/tests/test_scrollbar_colors.html:75:17
[task 2020-03-13T02:00:19.061Z] 02:00:19 INFO - Not taking screenshot here: see the one that was previously logged
[task 2020-03-13T02:00:19.061Z] 02:00:19 INFO - TEST-UNEXPECTED-FAIL | widget/tests/test_scrollbar_colors.html | Pixel count of color 0,0,255 - got 720, expected 1104
[task 2020-03-13T02:00:19.061Z] 02:00:19 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:383:14
[task 2020-03-13T02:00:19.062Z] 02:00:19 INFO - @widget/tests/test_scrollbar_colors.html:96:9
[task 2020-03-13T02:00:19.062Z] 02:00:19 INFO - Async*@widget/tests/test_scrollbar_colors.html:75:17
[task 2020-03-13T02:00:19.064Z] 02:00:19 INFO - Not taking screenshot here: see the one that was previously logged
[task 2020-03-13T02:00:19.064Z] 02:00:19 INFO - TEST-UNEXPECTED-FAIL | widget/tests/test_scrollbar_colors.html | Pixel count of color 0,255,255 - got 1180, expected 1152
[task 2020-03-13T02:00:19.065Z] 02:00:19 INFO - SimpleTest.is@SimpleTest/SimpleTest.js:383:14
[task 2020-03-13T02:00:19.066Z] 02:00:19 INFO - @widget/tests/test_scrollbar_colors.html:96:9
[task 2020-03-13T02:00:19.067Z] 02:00:19 INFO - Async*@widget/tests/test_scrollbar_colors.html:75:17
[task 2020-03-13T02:00:19.067Z] 02:00:19 INFO - GECKO(3033) | MEMORY STAT | vsize 130550498MB | residentFast 625MB
[task 2020-03-13T02:00:19.067Z] 02:00:19 INFO - TEST-OK | widget/tests/test_scrollbar_colors.html | took 698ms

Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/ab71da2b6acb
update pixel count expectations and re-enable test_scrollbar_colors.html for linux1804 r=jmaher
Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(egao)
Keywords: leave-open
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: