2027058 - For a11checks testruns, browser/components/profiles/tests/browser/browser_preferences.js is frequently orange with TEST-UNEXPECTED-PASS / "We expect at least one assertion to fail because this test file is marked as fail-if in the manifest."

Status: NEW

(Toolkit :: Startup and Profile System, defect)

Description

[Daniel Holbert [:dholbert]]

We marked the test browser/components/profiles/tests/browser/browser_preferences.js as failing in "a11y_checks" mode, here:
https://searchfox.org/firefox-main/rev/c836d51da4179d7b456d02d22e2f40c8b1a9b3d7/browser/components/profiles/tests/browser/browser.toml#62-65

["browser_preferences.js"]
fail-if = [
  "a11y_checks", # Bug 1955503
]

However, it apparently sometimes does not fail, and that's unexpected (per that annotation) and so that lack-of-failure is a TEST-UNEXPECTED-PASS (which then gets reported as a TEST-UNEXPECTED-FAIL and turns the task orange):
https://treeherder.mozilla.org/logviewer?job_id=555698942&repo=autoland&lineNumber=5412

[task 2026-03-26T01:14:07.930+00:00] 01:14:07     INFO - TEST-UNEXPECTED-PASS | browser/components/profiles/tests/browser/browser_preferences.js | We expect at least one assertion to fail because this test file is marked as fail-if in the manifest.
[task 2026-03-26T01:14:07.930+00:00] 01:14:07     INFO - TEST-INFO | expected FAIL
[task 2026-03-26T01:14:07.931+00:00] 01:14:07     INFO - GECKO(15798) | MEMORY STAT | vsize 11952MB | residentFast 725MB | heapAllocated 378MB
[task 2026-03-26T01:14:07.931+00:00] 01:14:07     INFO - checking for open sidebars
[task 2026-03-26T01:14:07.932+00:00] 01:14:07     INFO - checking window state
[task 2026-03-26T01:14:07.932+00:00] 01:14:07     INFO - TEST-INFO | started process screentopng
[task 2026-03-26T01:14:08.149+00:00] 01:14:08     INFO - TEST-INFO | screentopng: exit 0
[task 2026-03-26T01:14:08.149+00:00] 01:14:08     INFO - TEST-UNEXPECTED-FAIL | browser/components/profiles/tests/browser/browser_preferences.js | finished in 7195ms

Maybe we should skip the test on this platform, or mark it as random, instead of expected-failure? That would avoid these oranges.

This is one of the main failure-modes in intermittent bug 2005950. Note that in the intermittent failures view there, this category of failures is just reported as browser_preferences.js | finished in 7195ms (with whatever the test-duration was) - that's just the TEST-UNEXPECTED-FAIL line at the end of my code-quote above, pruned for brevity.

Comment 1

[Daniel Holbert [:dholbert]]

(In reply to Daniel Holbert [:dholbert] from comment #0)

Note that in the intermittent failures view there, this category of failures is just reported as browser_preferences.js | finished in 7195ms (with whatever the test-duration was) - that's just the TEST-UNEXPECTED-FAIL line at the end of my code-quote above, pruned for brevity.

(I filed bug 2027063 on possibly addressing this reporting quirk in the intermittent failures view.)

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:mossop, could you have a look please?

For more information, please visit BugBot documentation.