1894445 - Intermittent image/test/reftest/generic/moz-icon-blank-1-almostref.html == image/test/reftest/generic/moz-icon-blank-1-ref.html | single tracking bug

Status: RESOLVED INCOMPLETE

(Core :: Graphics: ImageLib, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=456527511&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TrJBQNyWRKaS49-kg50idw/runs/0/artifacts/public/logs/live_backing.log
Reftest URL: https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TrJBQNyWRKaS49-kg50idw/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

---

Comment 1

[Timothy Nikkel (:tnikkel)]

Odd, some sort of change caused this to happen? Maybe infra change or code change?

Comment 2

[Daniel Holbert [:dholbert]]

Not sure if it's related, but bug 1894564 also started happening recently and feels infra-ish, and sometimes these failures occur in the same tasks.

Comment 3

[Timothy Nikkel (:tnikkel)]

In the failing case the two images are blank pages with the top left corner folded over, but they are just slightly different icons.
When we pass the icons are identical, they are both pages with the top left corner folded over but with (illegible) text on the page. So it's not even wrong (given that it's just supposed to be an almostref), it's just a different result.

The tests just ask the os for icons and put up what the os returns.

Comment 4

[Daniel Holbert [:dholbert]]

(In reply to Timothy Nikkel (:tnikkel) from comment #1)

Odd, some sort of change caused this to happen? Maybe infra change or code change?

Bug 1888460 was a somewhat-recent infra change that might be relevant here, FWIW (just ran across it when bisecting the other bug that I'm looking at; see bug 1894564 comment 3).

Comment 5

[Daniel Holbert [:dholbert]]

tl;dr: The test probably just needs a fuzzy annotation for linux, similar to the existing fuzzy annotation for Windows and mac.

Currently it's expecting the testcase/reference to precisely match, only on linux, but that doesn't really make sense because the testcase and reference case are asking for different icons. (Maybe/probably on our test runners, the icon set used to be relatively-limited and the icons were in fact the same, at least when requested at the 16px size that we're using here; but that's no longer the case, I guess?)

Personally when I load the testcase locally on linux, the testcase and reference case render with different icons for me:

• The testcase has moz-icon://.txt?size=16 which for me is an icon that shows a document with lines of writing.
• The reference has moz-icon://bogus-unrecognized-icon.bogusunknown845?size=16 which renders similarly except with a question-mark instead of the lines of writing.
• So: they're two completely different (albeit similar) system icons, I think.

It looks like this test dates back to the patch in bug 1690845 (and it's been annotated as fuzzy-on-mac/windows as of when it landed, presumably due to an analogous difference on those platforms to what I'm seeing locally on Linux). I guess it never differed on our Linux test runners, but some infra change (possibly Bug 1888460) made us get a slightly different set of system-icons that are available (sometimes at least)?

Anyway: given that the icons are still similar and still look ~reasonable for the corresponding file types (they're not wrong in a way that users might consider buggy as in bug 1690845), I'd suggest we just add a fuzzy annotation for linux, to allow for what we're seeing.

Comment 6

[Timothy Nikkel (:tnikkel)]

Yes, the test does not expect an identical match, the naming almostref indicates it just wants to test that something reasonable is drawn. So if this was a consistent failure I'd be happy to update the fuzz. The fact that the OS returns different icons to us intermittently though suggests that some aspect of the configuration of these machines is changing on an intermittent basis. If the returned icons are changing what else about the config could be changing? This would make it hard to get consistent results on these machines when writing and running tests.

Comment 7

[Intermittent Failures Robot]

10 failures in 4059 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 2
• try: 1
• mozilla-beta: 2

Platform and build breakdown:

• linux1804-64-tsan-qr: 5
  • opt: 5
• linux1804-64-qr: 4
  • debug: 4
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1894445&startday=2024-04-29&endday=2024-05-05&tree=all

Comment 8

[Daniel Holbert [:dholbert]]

I think this bug is a manifestation of the strangely inconsistent choice-of-GTK-theme discussed in Bug 1895092; see that bug for more details.

Comment 9

[Intermittent Failures Robot]

10 failures in 3649 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• try: 5
• mozilla-release: 2

Platform and build breakdown:

• linux1804-64-qr: 5
  • debug: 5
• linux1804-64-tsan-qr: 4
  • opt: 4
• linux1804-64-asan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1894445&startday=2024-05-06&endday=2024-05-12&tree=all

Comment 10

[Intermittent Failures Robot]

38 failures in 4547 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 4
• mozilla-beta: 32
• try: 1
• autoland: 1

Platform and build breakdown:

• linux1804-64-asan-qr: 15
  • opt: 15
• linux1804-64-tsan-qr: 9
  • opt: 9
• linux1804-64-devedition-qr: 4
  • opt: 4
• linux1804-64-qr: 9
  • debug: 9
• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1894445&startday=2024-05-13&endday=2024-05-19&tree=all

Comment 11

[Intermittent Failures Robot]

38 failures in 3995 pushes (0.01 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-beta: 35
• mozilla-release: 3

Platform and build breakdown:

• linux1804-64-asan-qr: 8
  • opt: 8
• linux1804-64-tsan-qr: 11
  • opt: 11
• linux1804-64-qr: 12
  • debug: 12
• linux1804-64-shippable-qr: 2
  • opt: 2
• linux1804-64-devedition-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1894445&startday=2024-05-20&endday=2024-05-26&tree=all

Comment 12

[Intermittent Failures Robot]

11 failures in 4202 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 11

Platform and build breakdown:

• linux1804-64-asan-qr: 8
  • opt: 8
• linux1804-64-shippable-qr: 1
  • opt: 1
• linux1804-64-tsan-qr: 1
  • opt: 1
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1894445&startday=2024-05-27&endday=2024-06-02&tree=all

Comment 13

[Intermittent Failures Robot]

2 failures in 4577 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr115: 2

Platform and build breakdown:

• linux1804-64-tsan-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1894445&startday=2024-06-17&endday=2024-06-23&tree=all

Comment 14

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.