1809358 - test_scrollbar_colors.html fails on windows 11

Status: RESOLVED FIXED

(Core :: Widget: Win32, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

currently we run windows tests on windows10x64 for both 32bit and 64 bit builds of firefox.

I am working (in bug 1794900) to migrate tests to run on windows 11x64 instead, specifically 22H2 update (October 2022 major update).

As you can see on a try push, this specific test fails everywhere:
https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=d1433bf8663418bb320b6f2ca60494cfc70e6ad6&searchStr=mochitest-plain&selectedTaskRun=CONuVN5ASSSjDW7v-rImcQ.0

[task 2023-01-09T22:48:32.357Z] 22:48:32     INFO - TEST-START | widget/tests/test_scrollbar_colors.html
[task 2023-01-09T22:48:32.431Z] 22:48:32     INFO - GECKO(7112) | [Child 4984, Main Thread] WARNING: '!scrollbar', file /builds/worker/checkouts/gecko/widget/Theme.cpp:1105
[task 2023-01-09T22:48:32.481Z] 22:48:32     INFO - TEST-INFO | started process screenshot
[task 2023-01-09T22:48:34.598Z] 22:48:34     INFO - TEST-INFO | screenshot: exit 0
[task 2023-01-09T22:48:34.612Z] 22:48:34     INFO - Buffered messages logged at 22:48:32
[task 2023-01-09T22:48:34.612Z] 22:48:34     INFO - TEST-PASS | widget/tests/test_scrollbar_colors.html | Pixel count of color 255,255,0 
[task 2023-01-09T22:48:34.613Z] 22:48:34     INFO - Buffered messages finished
[task 2023-01-09T22:48:34.613Z] 22:48:34     INFO - TEST-UNEXPECTED-FAIL | widget/tests/test_scrollbar_colors.html | Pixel count of color 0,0,255 - got 324, expected 612
[task 2023-01-09T22:48:34.613Z] 22:48:34     INFO -     SimpleTest.is@SimpleTest/SimpleTest.js:487:14
[task 2023-01-09T22:48:34.613Z] 22:48:34     INFO -     @widget/tests/test_scrollbar_colors.html:118:7
[task 2023-01-09T22:48:34.614Z] 22:48:34     INFO - Not taking screenshot here: see the one that was previously logged
[task 2023-01-09T22:48:34.615Z] 22:48:34     INFO - TEST-UNEXPECTED-FAIL | widget/tests/test_scrollbar_colors.html | Pixel count of color 0,255,255 - got 2787, expected 2355
[task 2023-01-09T22:48:34.615Z] 22:48:34     INFO -     SimpleTest.is@SimpleTest/SimpleTest.js:487:14
[task 2023-01-09T22:48:34.615Z] 22:48:34     INFO -     @widget/tests/test_scrollbar_colors.html:118:7
[task 2023-01-09T22:48:34.616Z] 22:48:34     INFO - GECKO(7112) | MEMORY STAT | vsize 2111976MB | vsizeMaxContiguous 65100458MB | residentFast 109MB | heapAllocated 15MB
[task 2023-01-09T22:48:34.616Z] 22:48:34     INFO - GECKO(7112) | [Child 4984, Main Thread] WARNING: '!scrollbar', file /builds/worker/checkouts/gecko/widget/Theme.cpp:1105
[task 2023-01-09T22:48:34.617Z] 22:48:34     INFO - TEST-OK | widget/tests/test_scrollbar_colors.html | took 150ms

I am not sure if we need to adjust anything at the OS level, or at the test level. I see in the test we are failing at the WIN_NNT_REFERENCES section:
https://searchfox.org/mozilla-central/source/widget/tests/test_scrollbar_colors.html#72

I would like to figure out the right solution here- if there is non, we could skip this on all windows.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

:spohl, do you have thoughts on what to do here? If not, could you redirect to someone who might?

Comment 2

[Stephen A Pohl [:spohl]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #1)

:spohl, do you have thoughts on what to do here? If not, could you redirect to someone who might?

I'm pretty sure this is just an instance of scrollbars having a different size on Win11, therefore the pixel count will be different. Do we expect to have to run this test on both Win10 and Win11 for a while?

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I was planning on it, but if that is difficult, I could move some tests and others can run in parallel. There is a screenshot, although I wasn't sure how to judge that compared to what we have on win10

Comment 4

[Stephen A Pohl [:spohl]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #3)

I was planning on it, but if that is difficult, I could move some tests and others can run in parallel. There is a screenshot, although I wasn't sure how to judge that compared to what we have on win10

I'll take a look tomorrow. Can you point me to the screenshot that you mentioned? I couldn't find it.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

ack, thanks for calling me out on that- I was thinking of another bug where the screenshot helped out- the current screenshots don't show anything interesting.

to run the tests on win11, I do:
./mach try --worker-override='win10-64-2004=gecko-t/win11-64-2009-beta -q 'test-windows10' widget/tests/

Comment 6

[Stephen A Pohl [:spohl]]

Assigning myself since I'm working on a fix.

Comment 7

[Stephen A Pohl [:spohl]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #5)

ack, thanks for calling me out on that- I was thinking of another bug where the screenshot helped out- the current screenshots don't show anything interesting.

to run the tests on win11, I do:
./mach try --worker-override='win10-64-2004=gecko-t/win11-64-2009-beta -q 'test-windows10' widget/tests/

Hmm, this didn't seem to work for me. I adjusted this slightly because I'm trying to move the test from a mochitest to a browser-chrome mochitest:

./mach try --worker-override='win10-64-2004=gecko-t/win11-64-2009-beta -q 'test-windows10' widget/tests/browser/'

Try run:
https://treeherder.mozilla.org/jobs?repo=try&revision=712e3687ec93d43294c32fb25d07817eef9c5052&selectedTaskRun=ajfUtSFQQ-eAkl5tw-1dQg.0

What am I doing wrong..?

(fyi, the actual patch that I submitted is a wip patch that succeeds for me locally and I just wanted to test that I will be able to test the final patch on try)

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I had a typo and forgot a closing single quote, it should look like this:
./mach try --worker-override='win10-64-2004=gecko-t/win11-64-2009-beta' -q 'test-windows10' widget/tests/browser/

Comment 9

[Stephen A Pohl [:spohl]]

(In reply to Joel Maher ( :jmaher ) (UTC -8) from comment #8)

I had a typo and forgot a closing single quote, it should look like this:
./mach try --worker-override='win10-64-2004=gecko-t/win11-64-2009-beta' -q 'test-windows10' widget/tests/browser/

I tried this first, but I couldn't even get it to submit to try:

./mach try --worker-override='win10-64-2004=gecko-t/win11-64-2009-beta' -q 'test-windows10' widget/tests/browser
usage: mach [-h] [--strategy STRATEGY] [--tasks-regex TASKS_REGEX] [--tasks-regex-exclude TASKS_REGEX_EXCLUDE] [-m [MESSAGE]] [--closed-tree]
            [--stage-changes | --no-push] [--artifact | --no-artifact] [--env ENV] [--chemspill-prio] [--disable-pgo]
            [--worker-override WORKER_OVERRIDES] [--worker-suffix WORKER_SUFFIXES]
mach: error: unrecognized arguments for try auto: '-q', 'test-windows10', 'widget/tests/browser'

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

ack, I overlooked the need for fuzzy. I was taking recent command lines and copy/pasting stuff to get the --worker-override and somehow overlooked ./mach try fuzzy at the beginning.

Comment 11

[Stephen A Pohl [:spohl]]

Attachment: Bug 1809358: Make the test_scrollbar_colors.html mochitest a browser-chrome mochitest and update it with expected values to be run successfully on Windows 11. r=rkraesig

Comment 12

[Stephen A Pohl [:spohl]]

Thank you. I got this to pass now:

https://treeherder.mozilla.org/jobs?repo=try&revision=57a10e4007353c7b301a608039128ec5c3abeb8f&selectedTaskRun=APPTSJN2QEm62ZHNW4wa_A.0

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/398ef8a6698e
Make the test_scrollbar_colors.html mochitest a browser-chrome mochitest and update it with expected values to be run successfully on Windows 11. r=rkraesig,jmaher

Comment 14

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/398ef8a6698e

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:spohl, is this bug important enough to require an uplift?

• If yes, please nominate the patch for beta approval.
• If no, please set status-firefox110 to wontfix.

For more information, please visit auto_nag documentation.

Comment 16

[Stephen A Pohl [:spohl]]

Redirecting n-i.

Comment 17

[Joel Maher ( :jmaher ) (UTC -8)]

no need to uplift,