Open Bug 1912571 Opened 10 months ago Updated 4 months ago

Frequent selection/disabled-1.html == selection/disabled-1-ref.html | single tracking bug

Categories

(Core :: DOM: Selection, defect, P5)

Product:
Core
Component:
DOM: Selection
Type:
defect

Tracking

()

Status:
REOPENED

People

(Reporter: intermittent-bug-filer, Unassigned)

References

(Blocks 1 open bug)

Details

(Keywords: intermittent-failure, intermittent-testcase, leave-open, Whiteboard: [stockwell infra])

Attachments

(2 files)

Bug 1912571: Skip reftest selection/disabled-1.html on Android emulator for frequent failures. r?botond,hiro
48 bytes, text/x-phabricator-request
Details | Review
Bug 1912571: Skip reftest selection/disabled-1.html on Android emulator for frequent failures. r?botond,hiro
48 bytes, text/x-phabricator-request
phab-bot
: approval-mozilla-beta+
Details | Review

Filed by: smolnar [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=469960920&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Y1sh2sZwSdCZ9OHpQX0NNg/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Y1sh2sZwSdCZ9OHpQX0NNg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/reftests/selection/disabled-1-ref.html | 1400 / 2141 (65%)
[task 2024-08-10T07:52:32.885Z] 07:52:32     INFO -  REFTEST INFO | REFTEST fuzzy test (0, 0) <= (255, 15080) <= (3, 13)
[task 2024-08-10T07:52:32.885Z] 07:52:32  WARNING -  REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/selection/disabled-1.html == layout/reftests/selection/disabled-1-ref.html | image comparison, max difference: 255, number of differing pixels: 15080
[task 2024-08-10T07:52:32.924Z] 07:52:32     INFO -  REFTEST   IMAGE 1 (TEST):
INFO -  REFTEST   IMAGE 2 (REFERENCE):
INFO -  REFTEST TEST-END | layout/reftests/selection/disabled-1.html == layout/reftests/selection/disabled-1-ref.html

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.

Status: NEW → RESOLVED
Closed: 9 months ago
Resolution: --- → INCOMPLETE

New failure instance: https://treeherder.mozilla.org/logviewer?job_id=477945728&repo=mozilla-central

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---

There seems to be some sort of corruption going on in at least some of the test failures here.

Some of them show a testcase-screenshot from a completely different testcase, whose content is "Hello world" and CJK text, e.g. the failure in comment 0 and this more recent failure here:
https://treeherder.mozilla.org/logviewer?job_id=484827723&repo=autoland&lineNumber=8332
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/BOArJkyfQTmss2MuG8lg7w/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

I found another one that has a blob of green artifacts further down in the page:
https://treeherder.mozilla.org/logviewer?job_id=484830658&repo=mozilla-beta&lineNumber=8396
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Lp0m9gpOS-eJxqCzxfaFfQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

And another one where the reftest screenshot is entirely blank:
https://treeherder.mozilla.org/logviewer?job_id=484786811&repo=autoland&lineNumber=8343
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DLoZgRtBSBqu_iV3YTtbFw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

So: I'm suspicious that these are harness/infra issues rather than layout bugs. (Or possibly bugs in whatever part of our graphics code that we're using to extract the screenshot here.)

Andrew, could you please check what Daniel said in comment 11? These are still happening on android emulator devices, for eg:
https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/RGQqFbuRRdWPde-GrThrJQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1
Is there anything that can be done on the infra side of things?

Flags: needinfo?(aerickson)
Whiteboard: [stockwell disable-recommended] → [stockwell infra]

It looks like these jobs run on gecko-t/t-linux-kvm-noscratch-gcp which runs the handbuilt-docker-worker-tester-20240614 image. It's an older image using Ubuntu 16.04 and an older version of Docker. We have newer images based on 24.04 and a newer Docker version that might help (if it's a Docker issue). If you'd like to try running on a newer base image I'd talk to releng/jcristau (heading up the migration effort).

Not sure what the test image is (this runs in Docker, image is in tree). You could try using a newer base image or upgrading packages (the bug could be at that level). Nobody owns images in tree as far as I know (just whoever was the last committer). You can always add a new image also.

The android emulator and associated bits are built by an in-tree task, but it hasn't been updated in awhile. Joel and I have been talking about upgrading it in Q1. That could fix the issue also.

Flags: needinfo?(aerickson)
Summary: Intermittent selection/disabled-1.html == selection/disabled-1-ref.html | single tracking bug → Frequent selection/disabled-1.html == selection/disabled-1-ref.html | single tracking bug

The two most recent failures here...
https://treeherder.mozilla.org/logviewer?job_id=491597850&repo=autoland
https://treeherder.mozilla.org/logviewer?job_id=491595906&repo=autoland

...seem to be "real" failures, not the artifacts or wrong-snapshot issue mentioned in comment 11. In these failures, the mismatching pixels are simply the presence of a scrollbar. (In the first log above, the testcase has a ~short scrollbar visible and the reference case has none. In the second log above, the reference case has a ~long scrollbar visible and the testcase has none.)

That's probably responsible for a lot of the failures here, and we should look into that to see what might be going on there (I can do so tomorrow/Friday; it's late for me right now). ni=me as a reminder.

(I slightly wonder if that's the real issue in ~all of the failures here, even the ones that had the wrong screenshot reported, and in those cases we were detecting a real failure but somehow reporting the wrong screenshot due to some weird harness bug?)

Flags: needinfo?(dholbert)

as a shot in the dark, I wouldn't be surprised if adding this to the testcase and reference case will fix this up:

<meta name="viewport" content="width=device-width, initial-scale=1" />

The current test does show unnecessary-looking-scrollbars for me if I view it in responsive design mode at various small-ish scrollport sizes, and I think those go away if I add that viewport meta-tag.

(In reply to Daniel Holbert [:dholbert] from comment #25)

as a shot in the dark, I wouldn't be surprised if adding this to the testcase and reference case will fix this up:

<meta name="viewport" content="width=device-width, initial-scale=1" />

That didn't help, unfortunately. I'm testing out a few other ideas to see if we can come up with clues here. (here and on bug 1908583 which is a similar-ish failure in the same directory)

See Also: → 1908583

I wonder if this is something do do with our logic for bringing up the software keyboard, which maybe makes us think that some portion of the viewport, even though it's not actually showing in this case in our reftest harness... So e.g. when we got to paint that portion of the viewport, we sometimes paint ~random garbage for the portion that we think is covered up by the keyboard. That would at least explain the green artifacts halfway down the page in comment 11; and it would explain the duplicated-content that we're seeing in bug 1908583 comment 29.

FWIW it looks like we could at least avoid the scrollbar-related failures here (which is most of them) by adding...

:root { overflow:hidden}

...on the testcase and reference case here.

Here's a try run with that change, where almost all of the oranges are a failure in another test (bug 1908583), and the only failure in this test is a case where we're completely blank (which is one of the failure modes here per end of comment 11, though it's a less common one.)

I'm tempted to add that as a short-term band-aid to reduce the volume here.

(In reply to Daniel Holbert [:dholbert] from comment #30)

I wonder if this is something do do with our logic for bringing up the software keyboard, which maybe makes us think that some portion of the viewport, even though it's not actually showing in this case in our reftest harness... So e.g. when we got to paint that portion of the viewport, we sometimes paint ~random garbage for the portion that we think is covered up by the keyboard. That would at least explain the green artifacts halfway down the page in comment 11; and it would explain the duplicated-content that we're seeing in bug 1908583 comment 29.

FWIW in a recent change Hiro added inputmode="none" to this test to avoid bringing up the software keyboard.

Ah, that's good to know - thanks!

I see we have inputmode="none" already in reftest 1478604.html, which hiro added here back in November -- that's the one where we're getting duplicated content per bug 1908583 comment 29. I guess that failure could conceivably be from the keyboard having come up during trailing-space-1.html which is an earlier test and is one of the fixes in hiro's patch that you mentioned.

Let's see if the recent change that you referenced gives us any improvement here.... Hope so!

Flags: needinfo?(dholbert)

(In reply to Daniel Holbert [:dholbert] from comment #34)

I see we have inputmode="none" already in reftest 1478604.html, which hiro added here back in November -- that's the one where we're getting duplicated content per bug 1908583 comment 29.

FWIW I just noticed a new-to-me failure mode here:
https://treeherder.mozilla.org/logviewer?job_id=492056511&repo=autoland&lineNumber=13167

Reftest disabled-1.html is failing here, because the top half of the screenshot is actually showing a copy of the previous test (or maybe its reference case), 1478604.html, and the bottom half of the screenshot is showing a correct copy of the rendering for disabled-1.html.

This is along the lines of the duplicated content that I described in bug 1908583 comment 29, but the problem in this particular failure is with the top half of the screenshot being bogus, and the bottom half being correct.

https://treeherder.mozilla.org/logviewer?job_id=491936657&repo=try&lineNumber=8332 is similar to the one that I described in comment 39, except the top half of the testcase screenshot is showing writing-mode.html (and the bottom half shows the expected disabled-1.html rendering).

https://treeherder.mozilla.org/logviewer?job_id=492527061&repo=autoland&lineNumber=1822 is almost the same, with a weird glitch showing a little bit of 1478604.html for the top few pixels of the screenshot. (i.e. it's showing a frankenstein mashup of 3 different testcases in a single screenshot)

https://treeherder.mozilla.org/logviewer?job_id=492450797&repo=mozilla-central&lineNumber=15 is similar too, with a lime stripe separating the upper and lower section.

Some other recent failures, e.g.
https://treeherder.mozilla.org/logviewer?job_id=491972096&repo=mozilla-release&lineNumber=8396
https://treeherder.mozilla.org/logviewer?job_id=492544330&repo=mozilla-release&lineNumber=8338
https://treeherder.mozilla.org/logviewer?job_id=492522287&repo=autoland&lineNumber=1874
https://treeherder.mozilla.org/logviewer?job_id=492511474&repo=autoland&lineNumber=8290
... have the testcase only showing writing-mode.html, without anything on the bottom half.

Maybe around half of the failures here are ones where a scrollbar is showing in the reference and/or testcase screenshot -- e.g. here:
https://treeherder.mozilla.org/logviewer?job_id=492561745&repo=autoland&lineNumber=8342

My overflow:hidden hack in comment 31 would help with those, but that's still not a great band-aid because (a) we don't understand why that scrollbar is showing up intermittently and (b) we're still left with all the remaining failure volume, which is still above a tolerable level.

So, I suspect we should just skip this reftest on this platform for the time being.... hiro or botond, do you have any other theories or suggestions here?

A hypothesis I can think of why the scrollbar is there is the software keyboard which appeared in a test (bug 1918577) somehow affects there. In bug 1937785 comment 17 I saw a bunch of test failures due to the scrollbar appearance and it was mitigated by setting inputmode=none in a bunch of suspicious reftest which run before the failed reftests.

I'd say we should fix bug 1918577 and then see whether there remains still suspicious failures or not.

I have no idea about the half baked previous test screenshot, the half make me suspicious it's also related to the software keyboard.

See Also: → 1918577

Since it does not look like a fix for bug 1918577 is imminent, and this test is failing at a fairly high rate, I think it would be reasonable to disable the test on Android for now.

Assignee: nobody → dholbert
Assignee: dholbert → nobody
Pushed by dholbert@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/816d4da5fd04 Skip reftest selection/disabled-1.html on Android emulator for frequent failures. r=hiro
Assignee: nobody → dholbert
Assignee: dholbert → nobody
Attachment #9465764 - Flags: approval-mozilla-beta?

I filled out beta-uplift-form on bug 1908583 comment 40 (hopefully that works for this too, since I've got them stacked).

This is just a test annotation change, so hopefully it's uncontroversial. :)

Attachment #9465764 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: