Open Bug 1777050 Opened 3 years ago Updated 17 days ago

Intermittent layout/base/tests/test_reftests_with_caret.html | single tracking bug

Categories

(Core :: Layout, defect, P3)

Product:
Core
Component:
Layout
Type:
defect

Tracking

()

People

(Reporter: jmaher, Unassigned)

References

(Depends on 2 open bugs)

Details

(Keywords: intermittent-failure, intermittent-testcase, leave-open, Whiteboard: [retriggered])

Attachments

(1 file, 2 obsolete files)

Bug 1777050 - Disable some subtests in test_reftests_with_caret.html for frequent failures. r=#layout
48 bytes, text/x-phabricator-request
Details | Review
No description provided.

Additional information about this bug failures and frequency patterns can be found by running: ./mach test-info failure-report --bug 1777050

There have been 33 total failures in the last 7 days.
There are:

  • 1 failure on android-em-7-0-x86_64-lite-qr opt
  • 1 failure on android-em-7-0-x86_64-qr debug
  • 7 failures on macosx1015-64-qr opt and debug
  • 3 failures on macosx1015-64-shippable-qr opt
  • 6 failures on windows11-32-2009-qr opt
  • 6 failures on windows11-32-2009-shippable-qr opt
  • 7 failures on windows11-64-2009-qr opt
  • 2 failures on windows11-64-2009-shippable-qr opt

Recent failure log.

Daniel, as the owner of this component, can you help us assign the bug to someone? Thank you.

Flags: needinfo?(dholbert)
Whiteboard: [retriggered] → [retriggered][stockwell needswork:owner]

Looking at the reftest screenshots in a recent failure...
https://treeherder.mozilla.org/logviewer?job_id=428368535&repo=autoland&lineNumber=16269
... it looks like the testcase screenshot is lacking focus -- the caret isn't visible, and the focus-rect isn't drawn.

testcase screenshot
reference case screenshot

...and that's is the same symptom mentioned in bug 1741607 comment 2 (a prior version of this bug).

See that comment, bug 1741607 comment 10, and bug 1741607 comment 14 for more thoughts on this.

tl;dr it seems our focus() invocation in this testcase isn't always working.

In that older bug, I was thinking this was Windows-only, but at this point we've also got macOS and Android failures.

In a recent macOS failure log, there are a handful of failures; some of them show the textfield rendering with a focus outline but no caret; and some of them show some not-at-all-highlighted text in the testcase vs. highlighted text (i.e. selected text) in the reference case. These failure modes are more subtle and don't look like "the window just failed to get focus" like the Windows log from comment 66 and as discussed in bug 1741607 comment 2.

Flags: needinfo?(dholbert)

And in a recent Android failure log, there's a single subtest that's failing only because the textfield is scrolled (or snapped?) to a slightly different position in the testcase vs. reference case screenshot (off by 1px, it looks like). That one has no relevance to focus (or lack thereof) like the windows failures, or caret/highlight showing-up-or-not like the macOS failures.

For that Android failure:
testcase screenshot
reference case screenshot

So I think we have at least four different failure modes here (no focus; focus but caret is missing; missing highlight; and 1px-scroll-position-difference). Some of these may be specific to particular subtests (e.g. I'm noticing a lot of the Windows failures seem to be in bug923376.html).

Maybe some of these could be addressed by waiting slightly longer before taking the reftest screenshot? Hard to say.

It's not entirely clear that this "mochitest harness for a pile of reftests" still serves a useful purpose. I think it dates from a time when reftests weren't able to individually set about:config prefs or request focus, but they can do both of those now (via reftest.list and/or SpecialPowers).

So probably all (or most?) of the reftest "subtests" here could just be converted to "real" reftests at this point. And then if they continue to fail in these same ways, we can track and investigate the failures a bit more directly.

Depends on: 1852196

(In reply to Daniel Holbert [:dholbert] from comment #70)

So probably all (or most?) of the reftest "subtests" here could just be converted to "real" reftests at this point.

I spun off bug 1852196 to do that. I think that's the most valuable use of our time for making improvements here. If things work out, then test_reftests_with_caret.html will no longer exist when bug 1852196 is complete.

Hmm, it might actually be a bit tricky to convert these to "real" reftests; some of them do things like synthesizing keypresses with EventUtils.js; maybe we can do that in reftests these days (not sure) but I think it requires some reworking/rewriting.

So: maybe worth some papering-over / targeted fixes here after all.

Most of the failures here are on Windows, and most of those seem to be for bug923376.html as noted above, and to a lesser extent bug512295-2.html e.g. here:
https://treeherder.mozilla.org/logviewer?job_id=428442979&repo=autoland&lineNumber=16205
https://treeherder.mozilla.org/logviewer?job_id=428366870&repo=autoland&lineNumber=16689
https://treeherder.mozilla.org/logviewer?job_id=428366869&repo=autoland&lineNumber=16662

So maybe there's something we can do to address those failures, or (worst-case) we might consider disabling those subtests on Windows, as we've done with platform-specific-disablings for other subtests in this file.

Depends on: 1852691
No longer depends on: 1741607
Depends on: 1852821
Depends on: 1853958

I think bug 1853958 will remove a substantial portion of the intermittent-failure volume here (in particular, the Windows failures mentioned in comment 73 for subtests bug923376.html and bug512295-2.html).

Attachment #9386384 - Attachment is obsolete: true

Updates:
There have been 48 total failures in the last 7 days.
There are:

  • 10 failures on android-em-7-0-x86_64-qr opt
  • 4 failures on linux1804-64-shippable-qr opt
  • 11 failures on macosx1015-64-qr opt
  • 2 failures on macosx1015-64-shippable-qr opt
  • 1 failure on windows10-64-2009-qr opt
  • 4 failures on windows11-32-2009-qr opt
  • 6 failures on windows11-32-2009-shippable-qr opt
  • 1 failure on windows11-64-2009-qr opt
  • 9 failures on windows11-64-2009-shippable-qr opt

Recent failure log.

Keywords: leave-open
Flags: needinfo?(emilio)
Flags: needinfo?(emilio)
Pushed by ealvarez@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/cc613715aed4 Disable some subtests in test_reftests_with_caret.html for frequent failures. r=dholbert
Attachment #9424868 - Attachment is obsolete: true
Whiteboard: [retriggered][stockwell disable-recommended] → [retriggered]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: