2002097 - Intermittent [tier 2] /webdriver/tests/bidi/browsing_context/create/background.py | test_background[False-tab] - AssertionError: assert 'hidden' == 'visible'

Status: RESOLVED INCOMPLETE

(Remote Protocol :: WebDriver BiDi, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=537226577&repo=mozilla-central&task=W9L5VZ3QSGW9ULo8VH5L-w.1
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/W9L5VZ3QSGW9ULo8VH5L-w/runs/1/artifacts/public/logs/live_backing.log

---

[task 2025-11-24T20:02:55.262+00:00] 20:02:55     INFO - TEST-PASS | /webdriver/tests/bidi/browsing_context/create/background.py | test_background[True-window] 
[task 2025-11-24T20:02:55.262+00:00] 20:02:55     INFO - TEST-UNEXPECTED-FAIL | /webdriver/tests/bidi/browsing_context/create/background.py | test_background[False-tab] - AssertionError: assert 'hidden' == 'visible'
[task 2025-11-24T20:02:55.262+00:00] 20:02:55     INFO - bidi_session = <webdriver.bidi.client.BidiSession object at 0x000001BCD741BBD0>
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - top_context = {'children': [], 'clientWindow': 'db3a7ba7-040e-42e8-af44-f5c9c37ae6f3', 'context': 'fdfde9e9-77db-4441-97f9-b1fc34f75840', 'originalOpener': None, ...}
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - type_hint = 'tab', background = False
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - 
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -     @pytest.mark.parametrize("type_hint", ["tab", "window"])
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -     @pytest.mark.parametrize("background", [True, False])
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -     async def test_background(bidi_session, top_context, type_hint, background):
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -         new_context = await bidi_session.browsing_context.create(type_hint=type_hint, background=background)
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -     
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -         try:
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -             if background:
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -                 assert await get_visibility_state(bidi_session, top_context) == "visible"
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO -             else:
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - >               assert await get_visibility_state(bidi_session, new_context) == "visible"
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - E               AssertionError: assert 'hidden' == 'visible'
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - E                 
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - E                 - visible
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - E                 + hidden
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - 
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - background = False
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - bidi_session = <webdriver.bidi.client.BidiSession object at 0x000001BCD741BBD0>
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - new_context = {'context': '19aae95c-b194-4f77-90c3-d5d2c786c20d'}
[task 2025-11-24T20:02:55.263+00:00] 20:02:55     INFO - top_context = {'children': [],
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO -  'clientWindow': 'db3a7ba7-040e-42e8-af44-f5c9c37ae6f3',
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO -  'context': 'fdfde9e9-77db-4441-97f9-b1fc34f75840',
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO -  'originalOpener': None,
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO -  'parent': None,
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO -  'url': 'about:blank',
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO -  'userContext': 'default'}
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO - type_hint  = 'tab'
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO - 
[task 2025-11-24T20:02:55.264+00:00] 20:02:55     INFO - tests\web-platform\tests\webdriver\tests\bidi\browsing_context\create\background.py:28: AssertionError
[task 2025-11-24T20:02:55.265+00:00] 20:02:55     INFO - .
[task 2025-11-24T20:02:55.265+00:00] 20:02:55     INFO - TEST-OK | /webdriver/tests/bidi/browsing_context/create/background.py | took 9273ms

Comment 1

[Intermittent Failures Robot]

13 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 13

Table

	headless	no_variant
windows11-x86_64/ccov	6	7

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2002097&startday=2025-11-24&endday=2025-11-30&tree=all

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This started when the patch on bug 543435 landed. So I'm fairly sure this is related to the initial about:blank changes.

Vincent, might taking a look at it?

Comment 3

[Vincent Hilla [:vhilla]]

I remember bug 1976026 and how nothing guaranteed a new window is visible when it becomes active. Visibility on windows seemed racy and this test is probably another place where it shows up. This intermittent being windows-only so far aligns with that. This also only shows up with slow CCOV tasks right now.

Here's the follow-up I filed back then about visibility issues: bug 1978353.

Comment 4

[Vincent Hilla [:vhilla]]

:emilio we talked about this in the context of tab switching. I vaguely remember that OS timing is involved, visbility gets updated async and that we don't completely control whether a window becomes visible. It could be occluded or another window could be opened around the same time that gets focused instead.

Do you know someone that is familiar with windows widgets and could take a look? Or do we continue to accept visibility to be racy and change the test expectations for windows ccov?

Comment 5

[Emilio Cobos Álvarez [:emilio]]

The obvious questions here are:

• Can we see a failure screenshot? I think mochitests do have that ability, it'd be good to get the same on webdriver tests?
• Does this test start passing reliably if you set widget.windows.window_occlusion_tracking.enabled=false?

Comment 6

[Julian Descottes [:jdescottes]]

(In reply to Emilio Cobos Álvarez [:emilio] from comment #5)

The obvious questions here are:

• Can we see a failure screenshot? I think mochitests do have that ability, it'd be good to get the same on webdriver tests?

We can't at the moment but I agree this would be useful.

• Does this test start passing reliably if you set widget.windows.window_occlusion_tracking.enabled=false?

We already force this at https://searchfox.org/firefox-main/rev/5ccf4a7d77a329f237d3a41e400049f9c47dc71f/remote/shared/RecommendedPreferences.sys.mjs#406

["widget.windows.window_occlusion_tracking.enabled", false],

Comment 7

[Julian Descottes [:jdescottes]]

Let's see if waiting for document visibility in the command helps. baseline: https://treeherder.mozilla.org/jobs?repo=try&revision=93ac68c4e11e2ee2003774be4df9b158813ecf2f, with patch: https://treeherder.mozilla.org/jobs?repo=try&revision=e55c666960d6ddfbbf92f55d55b105d5685236ff

Edit: Cannot trigger the issue on try so far.

Comment 8

[Julian Descottes [:jdescottes]]

Going to clear the ni? for Henrik, who also mentioned we already have https://bugzilla.mozilla.org/show_bug.cgi?id=1447852 to add screenshots to wpt tests.

Comment 9

[Julian Descottes [:jdescottes]]

For some reason, the failure seems to be triggered much more regularly when running the full test suite: ~25% of failures on the wd4 job at https://treeherder.mozilla.org/jobs?repo=try&revision=52bb1f9f5a300a002c9e397bdca014442f1827b7, but no failure with my patch: https://treeherder.mozilla.org/jobs?repo=try&revision=b514472e527bd98ed48d27192cd3c1b8ae8d4b75

Comment 10

[Julian Descottes [:jdescottes]]

For now I'm proposing to additionally wait for the document to be visible in the command which creates new foreground tabs: Bug 2003857, is that fine with you or do you prefer to investigate this on the platform side and avoid the workaround?

Comment 11

[Vincent Hilla [:vhilla]]

I'm fine with workarounds. There is bug 1978353 to track the platform issue and this hasn't caused much problems yet.

Though for BrowserTestUtils.openNewBrowserWindow, waiting for visibility didn't work for two reasons.

1. Some tests opened multiple windows at once that didn't all become visible.
2. It's been months since I worked on this, so I might remember incorrectly. But at the point where I tried it in that method, the visibility event wasn't reliable. The window could've become visible before where I added the listener, or there could be three visibilityupdate (visible, hidden, visible) in short succession.

Comment 12

[Julian Descottes [:jdescottes]]

(In reply to Vincent Hilla [:vhilla] from comment #11)

I'm fine with workarounds. There is bug 1978353 to track the platform issue and this hasn't caused much problems yet.

Though for BrowserTestUtils.openNewBrowserWindow, waiting for visibility didn't work for two reasons.

1. Some tests opened multiple windows at once that didn't all become visible.
2. It's been months since I worked on this, so I might remember incorrectly. But at the point where I tried it in that method, the visibility event wasn't reliable. The window could've become visible before where I added the listener, or there could be three visibilityupdate (visible, hidden, visible) in short succession.

Thanks for the feedback Vincent.
At least in our case, we are not waiting for an event but polling until the document visibility is "visible" (I think I will rename our internal helper because #waitForVisibilityChange confused me as well). So window could've become visible before is not an issue, but the triple event could be. At the very least this seems to fix our intermittent, so I imagine it's good enough for now!

Comment 13

[Intermittent Failures Robot]

13 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 13

Table

	headless	no_variant
windows11-x86_64/ccov	6	7

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2002097&startday=2025-12-01&endday=2025-12-07&tree=all

Comment 14

[Julian Descottes [:jdescottes]]

Attachment: Bug 2002097 - [wdspec] Add helper to wait for visibility and focus

Comment 15

[Intermittent Failures Robot]

3 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Table

	headless	no_variant
windows11-x86_64/ccov	1	2

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2002097&startday=2025-12-08&endday=2025-12-14&tree=all

Comment 16

[BugBot [:suhaib / :marco/ :calixte]]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit BugBot documentation.