2048072 - Intermittent macosx1500-aarch64-vms testing/marionette/harness/marionette_harness/tests/unit/test_chrome_switch_window.py | test_switch_tabs_for_new_background_window_without_focus_change | AssertionError: 0 != 1

Status: NEW

(Testing :: Marionette Client and Harness, defect, P5)

Description

[Ryan Curran]

Intermittent failure of TestSwitchWindowChrome.test_switch_tabs_for_new_background_window_without_focus_change (testing/marionette/harness/marionette_harness/tests/unit/test_chrome_switch_window.py) on the headless macosx1500-aarch64-vms pool (Bug 2037084).

The test opens a new background window and expects get_selected_tab_index() to remain on the originally-selected tab. get_selected_tab_index() resolves the foreground window via BrowserWindowTracker.getTopWindow(), which returns the front of _trackedWindows -- a list ordered by the OS "activate" event. On the VM workers there is no window manager / WindowServer, so activation ordering races when a new (background) window is created and getTopWindow() can transiently return the new window, yielding the wrong selected-tab index. This is intermittent rather than permanent, consistent with a race on activation events.

Split off from tracking bug 1989258. Fix: skip the test on hosts without a window manager via a new skip_if_no_window_manager decorator (generalizable to other headless platforms).

Comment 1

[Ryan Curran]

Attachment: Bug 2048072 - Add skip_if_no_window_manager and use it for marionette tests that need a window manager r?whimboo

Add a skip_if_no_window_manager decorator (in marionette's decorators.py,
alongside skip_if_chrome) and apply it to
TestSwitchWindowChrome.test_switch_tabs_for_new_background_window_without_focus_change.

The test opens a new background window and expects get_selected_tab_index()
to stay on the originally-selected tab. get_selected_tab_index() resolves the
foreground window via BrowserWindowTracker.getTopWindow(), which returns the
front of _trackedWindows -- a list ordered by the OS "activate" event. On
headless macOS VM workers there is no window manager, so activation ordering
races when the background window is created and getTopWindow() can transiently
return the new window, intermittently yielding the wrong selected-tab index.

The decorator currently detects Apple Virtualization Framework guests and is
structured so other headless platforms can be added later.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hi Ryan, is there something remaining which is still blocking the landing of the patch?

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/ef46974b6a35
https://hg.mozilla.org/integration/autoland/rev/b4791af8ec43
Add skip_if_no_window_manager and use it for marionette tests that need a window manager r=whimboo

Comment 4

[Ryan Curran]

Try evidence for this intermittent (whimboo's request to reference the failure):

Diagnostic run with all three originally-affected tests enabled on macosx1500-aarch64-vms, rebuilt 6x:
https://treeherder.mozilla.org/jobs?repo=try&revision=b3d87eb02cc6
test_switch_tabs_for_new_background_window_without_focus_change PASSED on the VM in all 6 runs there.

A separate run caught the failure:
https://treeherder.mozilla.org/jobs?repo=try&revision=e32f53280584
TEST-UNEXPECTED-FAIL | testing/marionette/harness/marionette_harness/tests/unit/test_chrome_switch_window.py TestSwitchWindowChrome.test_switch_tabs_for_new_background_window_without_focus_change | AssertionError: 0 != 1

So it is a low-rate intermittent (1 failure in ~8 observed VM runs), consistent with the race you suspected: get_selected_tab_index() resolves the foreground window via BrowserWindowTracker.getTopWindow(), which returns the front of _trackedWindows -- a list ordered by the OS "activate" event. On the headless VM (no window manager) the activate ordering races when a background window is created, so getTopWindow() transiently returns the new window and the selected-tab index comes back 0 instead of the expected 1.

This test stays skipped on the VM (skip_if_no_user_session). The other two tests originally lumped under the VM skips -- test_click_raises_element_not_accessible (Bug 2047574) and test_switch_to_unloaded_tab (Bug 2048226) -- pass on the VM and are being re-enabled in D308058 / D308059.

Comment 5

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/b4791af8ec43

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Given that this is not permanently failing, it's a race. So one thing that we could actually test here is to wait for the correct window to be returned from the script execution command. Sadly that on macOS we do not support the widget:5 logs to see if the focus/active status changes at some point. So I need to think a bit which kind of log we might be able to use or manually add to get more details out of Firefox.

Comment 7

[Ryan Curran]

Agreed it's the getTopWindow/_trackedWindows activate-ordering race. Since it's ~1/8 on the VM, I'll run high-rebuild marionette-unittest try pushes on macosx1500-aarch64-vms to capture more failure instances, and I'm happy to push any build with extra logging you add and pull the artifacts from the VM run. If we'd rather not block on widget logging, a Wait(...).until() on the expected window in the test would paper over the race. Stays skipped via skip_if_no_user_session meanwhile.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Here a link to the BrowserWindowTracker.getTopWindow method, which has quite a lot of conditions that are getting checked:
https://searchfox.org/firefox-main/rev/c280a3de79038960073359b0b4777c676320b76a/browser/modules/BrowserWindowTracker.sys.mjs#207-246

I would start there and maybe dump() all the to be checked values additional to the tab count so that we can identify the window. Comparing a passing and failing runs should most likely give us an idea. Interestingly none of these checks are actually related to the focus / activation state.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hi Ryan, something is wrong with the skip patch as landed. The failures are still present and get classified for bug 1989258. I assume that the code for _running_without_user_session() is not doing what it should do?

https://searchfox.org/firefox-main/rev/542208dcdb891e1efa70d9411d612c0ee0726dd8/testing/marionette/harness/marionette_harness/marionette_test/decorators.py#29-35

Comment 10

[Ryan Curran]

You're right -- the skip isn't firing. In the verification run the test logs TEST-PASS, not TEST-SKIP, so _running_without_user_session() returns False on the VM: 'launchctl print gui/UID' succeeds because cltbld is logged into an Aqua session (needed to render tests), so the gui domain exists.

This is the same trap as the earlier WindowServer check. These headless VM workers run a WindowServer, have a logged-in GUI session, and even report a 1920x1080 framebuffer -- so by capability they're indistinguishable from bare metal. The getTopWindow/_trackedWindows activate race itself isn't queryable from the environment; the only reliable discriminator is that it is a VM.

Fix: detect the VM directly -- 'sysctl -n hw.model' startswith VirtualMac (self-contained, what the original helper used), or the macos_vm mozinfo flag (D302479) -- and rename skip_if_macos_vm. I know that is back toward the model check, but the VM genuinely mimics a real display environment, so capability-detection cannot work here. Patch incoming (r?whimboo).

Comment 11

[Ryan Curran]

Attachment: Bug 2048072 - Skip test_switch_tabs_for_new_background_window_without_focus_change by detecting the macOS VM, not a GUI session r?whimboo

The previous skip used skip_if_no_user_session (launchctl print gui/<uid>),
but the macOS VM workers run cltbld in a logged-in Aqua session to render
tests, so that launchd domain exists, the check returned False, and the skip
never fired -- the test kept failing intermittently (classified for bug
1989258).

These headless VM workers also run a WindowServer and report a framebuffer,
so by capability they are indistinguishable from bare metal; only the
hardware model identifies them. Detect the VM directly via
sysctl -n hw.model starting with "VirtualMac", and rename
skip_if_no_user_session -> skip_if_macos_vm (helper -> _running_on_macos_vm).

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/64efd68db913
https://hg.mozilla.org/integration/autoland/rev/59818d69464f
Skip test_switch_tabs_for_new_background_window_without_focus_change by detecting the macOS VM, not a GUI session r=whimboo

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Updating the summary to reflect that it's not the missing window manager which is causing it. Lets see when I'll find the time to check what's going on here.

Comment 14

[amarc]

https://hg.mozilla.org/mozilla-central/rev/59818d69464f

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Hi Ryan, on autoland/mozilla-central it looks all good now. Thanks! But the failures on mozilla-beta are quite high. Would you mind creating an uplift request so that we can skip the test there as well? Thanks!

I'm going to update the bug's summary as well to allow sheriffs to better classify the failure as long as it is still occurring.

Comment 16

[Ryan Curran]

Attachment: Bug 2048072 - Skip test_switch_tabs_for_new_background_window_without_focus_change by detecting the macOS VM, not a GUI session

The previous skip used skip_if_no_user_session (launchctl print gui/<uid>),
but the macOS VM workers run cltbld in a logged-in Aqua session to render
tests, so that launchd domain exists, the check returned False, and the skip
never fired -- the test kept failing intermittently (classified for bug
1989258).

These headless VM workers also run a WindowServer and report a framebuffer,
so by capability they are indistinguishable from bare metal; only the
hardware model identifies them. Detect the VM directly via
sysctl -n hw.model starting with "VirtualMac", and rename
skip_if_no_user_session -> skip_if_macos_vm (helper -> _running_on_macos_vm).

Original Revision: https://phabricator.services.mozilla.com/D309561

Comment 17

[Ryan Curran]

Attachment: Bug 2048072 - Skip test_switch_tabs_for_new_background_window_without_focus_change on macOS VM workers

Beta has none of the macOS-VM skip infrastructure that landed on central this
cycle, so add skip_if_macos_vm (detecting the VM via sysctl hw.model =
VirtualMac) and apply it to the racy test directly. This matches the final
central state from D309561.

Comment 18

[Phabricator Automation]

firefox-beta Uplift Approval Request

• User impact if declined/Reason for urgency: None — this is test-only. Without it, the marionette test test_switch_tabs_for_new_background_window_without_focus_change keeps intermittently failing on the macOS VM worker pools (macosx1500-aarch64-vms), where it races on native window activation. That produces ~41 recurring orange instances over the last 2 weeks on mozilla-beta (beta + devedition), all classifying under bug 1989258, adding CI noise and sheriff/retrigger overhead. No user-facing product impact.
• Code covered by automated testing?: yes
• Fix verified in Nightly?: yes
• Needs manual QE testing?: no
• Steps to reproduce for manual QE testing:
• Risk associated with taking this patch: low
• Explanation of risk level: Test-only. It adds a skip_if_macos_vm decorator (which detects the VM via sysctl -n hw.model starting with "VirtualMac") and applies it to a single marionette test. No product/runtime code is touched, and the guard only triggers on macOS VM workers — bare-metal and all other platforms are unaffected. Note: beta does not yet have any of the macOS-VM skip infrastructure (it landed on central this cycle), so this patch adds the helper + decorator from scratch rather than being a delta cherry-pick; the resulting state is byte-identical to central.
• String changes made/needed?: None
• Is Android affected?: no

Comment 19

[Intermittent Failures Robot]

7 failures were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• mozilla-beta: 6

Table

	no_variant
macosx1500-aarch64/opt	7

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2048072&startday=2026-06-29&endday=2026-07-05&tree=all

Comment 20

[Pulsebot]

https://github.com/mozilla-firefox/firefox/commit/00c784e13fb9
https://hg.mozilla.org/releases/mozilla-beta/rev/984d4ebef5ad

Comment 21

[Ryan Curran]

Beta uplift landed July 6 as 984d4ebef5ad (a=pascalc), central 00c784e13fb9. This skips the test on macOS VM workers via the skip_if_macos_vm decorator (VirtualMac model detection), so the beta noise should stop from the next beta build. Clearing needinfo

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thank you Ryan. For the remaining investigation and fix lets unassign you.

Comment 23

[Intermittent Failures Robot]

3 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-beta: 2

Table

	no_variant
macosx1500-aarch64/opt	3

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2048072&startday=2026-07-06&endday=2026-07-12&tree=all

Comment 24

[Intermittent Failures Robot]

1 failures were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Table

	no_variant
windows11-x86_64/asan	1

For more details, see:

https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=2048072&startday=2026-07-20&endday=2026-07-26&tree=all