Closed Bug 2034604 Opened 3 months ago Closed 3 months ago

run macOS mochitests on AArch64

Categories

(Firefox Build System :: Task Configuration, task)

task

Tracking

(firefox152 fixed)

RESOLVED FIXED
152 Branch
Tracking Status
firefox152 --- fixed

People

(Reporter: rcurran, Assigned: rcurran)

References

(Depends on 3 open bugs)

Details

Attachments

(20 files)

48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review
48 bytes, text/x-phabricator-request
Details | Review

To reduce the load on the macOS Intel CPU test machines, mochitests should be moved to the macOS AArch64 workers

Move mochitest suites from macosx1470-64-tests (Intel) to
macosx1500-aarch64-tests (Apple Silicon M4).

Skip-if annotations added for known ARM failures (Bug 2034604):

  • devtools/client/netmonitor browser_net_frame.js
  • devtools/server/tests browser_canvasframe_helper_04.js
  • devtools/client/framework browser_toolbox_meatball.js
  • devtools/client/inspector/rules browser_rules_variables-jump-to-definition.js
  • dom/events test_coalesce_touchmove.html, test_moz_mouse_pixel_scroll_event.html,
    test_wheel_zoom_on_form_controls.html
  • dom/canvas/test/webgl-conf tex-unpack-params.html (debug crash)
  • dom/credentialmanagement test_account_label.html
  • dom/security/test/mixedcontentblocker test_bug803225.html
  • dom/tests/mochitest/pointerlock test_pointerlock_xorigin_iframe_movementXY.html
  • gfx/layers/apz test_group_touchevents-5.html
  • accessible/tests/browser/tree browser_select.js
  • image/test/mochitest test_animation_operators.html
  • layout/style/test/chrome test_display_mode_reflow.html
  • toolkit/components/aboutmemory test_aboutmemory5.xhtml, test_aboutmemory7.xhtml
  • toolkit/content/tests/chrome test_menulist.xhtml
  • toolkit/content/tests/widgets test_image_recognition_zh.html

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

https://treeherder.mozilla.org/jobs?repo=try&revision=b89a6f3ad24334d891229f0982c7f5ee9ce6958c

Flags: needinfo?(aryx.bugmail)
Regressions: 2035975
Regressions: 2035978
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → 152 Branch

A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)

The test fails permanently if only the manifest which contains it runs on
macOS 15.30 AArch64 debug.

Attachment #9575528 - Attachment description: WIP: Bug 2034604 - skip browser_application_panel_open-links.js on macOS AArch64 as it had been on Intel → Bug 2034604 - skip browser_application_panel_open-links.js on macOS AArch64 as it had been on Intel
Pushed by archaeopteryx@coole-files.de: https://github.com/mozilla-firefox/firefox/commit/29df79939e78 https://hg.mozilla.org/integration/autoland/rev/12459654a67c skip browser_application_panel_open-links.js on macOS AArch64 as it had been on Intel DONTBUILD
Blocks: 2037064

Move six suites off Intel onto the M4 fleet: jsreftest, reftest,
marionette-integration, marionette-unittest, telemetry-tests-client,
test-verify-wpt. All validated clean on macosx1500-aarch64/{debug,opt}
across two validation pushes after the two fuzzy annotations below.

Reftest fuzzy annotations for Apple Silicon-specific pixel variance:

  • layout/reftests/box-shadow/boxshadow-rotated.html: M4 saw 5442
    differing pixels (max 13), one above the existing 5250 ceiling.
    Add fuzzy-if(appleSilicon,0-13,0-6000).
  • layout/reftests/bugs/1415987-1.html: M4 saw 6029 differing pixels
    (max 255), four above the existing 6025 ceiling. Existing entry
    already has a wider gtkWidget allowance; add a parallel
    fuzzy-if(appleSilicon,0-255,0-6500).

Both are software-rendering pixel-rasterization differences across
architectures, not regressions.

Initially included gtest, test-verify, and test-verify-gpu but dropped:

  • gtest hits arch-specific failures (PHC.TestPHCAllocations,
    Jemalloc.BaseAllocMergeSplit, MediaDataEncoderTest.H264Encode4K*,
    VP8VideoTrackEncoder.*, TestUDPSocket.TestUDPSocketMain). The PHC
    test has a pending fix on Bug 2021381; the encoder/jemalloc tests
    need their own evaluation. Moving gtest separately.
  • test-verify (debug+opt) fails in mozharness setup with
    ModuleNotFoundError: No module named 'yaml' coming out of
    manifestparser/toml.py. PyYAML is missing from the worker Python
    env on gecko-t-osx-1500-m4 in the verify-tools path. Doesn't affect
    test-verify-wpt, which uses a different mozharness path.
    test-verify-gpu likely has the same issue but wasn't scheduled in
    the validation push (no GPU test mods); deferring with test-verify.

crashtest is left alone here; it's covered by D294786 (Bug 2032601)
already in flight.

These were initially included in D299130 but dropped after the validation
try push surfaced a missing pyyaml in the M4 worker Python env, causing
mozharness setup to fail with ModuleNotFoundError: No module named 'yaml'
during find_tests_for_verification. test-verify-wpt was unaffected
(different mozharness path) and migrated as part of D299130.

ronin_puppet PR #1204 (mozilla-platform-ops/ronin_puppet#1204) added
packages::python3_yaml to the gecko-t-osx-1500-m4 role; merged 2026-05-07
and propagating across the M4 fleet via per-task reboots. Confirmed
working on macmini-m4-111: pip3 show pyyaml reports 6.0.2 post-Puppet.

With pyyaml in place, test-verify can now run on M4. Move it and
test-verify-gpu off macosx1470-64-tests onto macosx1500-aarch64-tests.

Pushed by archaeopteryx@coole-files.de: https://github.com/mozilla-firefox/firefox/commit/10ff8c9be80d https://hg.mozilla.org/integration/autoland/rev/01986c788157 Migrate low-risk test suites from macosx1470-64 to macosx1500-aarch64 r=aryx,layout-reviewers,emilio
Depends on: 2038643

jittest and xpcshell were added to macosx1500-aarch64-tests in D294786
(landed as dc3288403045 on autoland 2026-05-12). Remove them from
macosx1470-64-tests now that M4 is running them, completing the cutover
for those two suites and dropping their duplicated load from the Intel
fleet.

Convention per recent migration commits (D299130, D299163): each
add-to-aarch64 patch should be paired with a remove-from-Intel
follow-up so Intel hardware load actually drops.

crashtest is intentionally left on Intel for now. Validation try push
showed pre-existing crashtest failures on macosx1500-aarch64 that
predate this patch (autoland has been ~80% fail on debug-crashtest and
~94% fail on opt-crashtest since D294786 landed). The failing tests
include dom/media/webrtc/tests/crashtests/2038533.html and
editor/libeditor/crashtests/1429523.{html,xhtml}, 1444630.html, all
with the same reftest-wait timeout pattern. Keeping crashtest on Intel
preserves a stable signal until those failures are resolved
separately.

After this lands, Intel (macosx1470-64-tests) retains:

  • cppunittest (intentionally cross-arch; stays)
  • crashtest (held pending M4 stability)
  • firefox-ui-functional (single-suite eval pending)
  • gtest (deferred; needs arch-specific failure triage)
  • 7 web-platform-tests sub-suites (separate batch)
Depends on: 2039098

These are the two lowest-risk WPT sub-suites for the Sonoma Intel ->
Sequoia M4 migration: wdspec is WebDriver-protocol-driven with no
rendering dependency, and crashtest is "page loaded without crashing"
assertions with minimal rendering reliance.

This patch adds them to the hardware M4 test platform only
(macosx1500-aarch64-tests). They are intentionally not added to the
VM test set (macosx1500-aarch64-vms-tests) per the existing comment
block at that set: Bug 2038639 (GPU process leak in
APZCTreeManager::CheckerboardFlushObservers on headless macOS) blocks
web-platform-tests coverage on the VM platform. Once that bug
resolves, a follow-up can extend wdspec + crashtest to VMs.

This is the add-half of the cutover pattern established by D294786 /
D300150: add to M4, validate, then a follow-up patch removes the
sub-suites from macosx1470-64-tests.

These two WPT sub-suites were added to macosx1500-aarch64-tests in
D301506 (landed as 7a1a0928aceb on autoland 2026-05-20). Remove them
from macosx1470-64-tests now that the M4 platform is running them,
completing the cutover for WPT Batch 1 and dropping the duplicated
load from the Intel fleet.

Convention per recent migration commits (D294786 + D300150, D301506):
each add-to-aarch64 patch should be paired with a remove-from-Intel
follow-up so Intel hardware load actually drops.

After this lands, Intel (macosx1470-64-tests) retains:

  • cppunittest (intentionally cross-arch; stays)
  • crashtest (held pending Bug 2039098 fix)
  • firefox-ui-functional (single-suite eval pending)
  • gtest (deferred; needs arch-specific failure triage)
  • 5 web-platform-tests sub-suites still to be batched:
    canvas, eme, print-reftest, reftest, webcodecs

WPT Batch 2 — same pattern as D301506 (web-platform-tests-crashtest +
-wdspec) and the earlier non-WPT cutovers (D294786 + D300150).

canvas tests exercise the 2D rendering API; software-rasterizable in
practice, no GPU dependency required. webcodecs covers codec encode
and decode; the CPU codec path is used by default, hardware codec
behavior is not asserted by the WPT suite.

Both sub-suites are intentionally added to the hardware M4 test set
only (macosx1500-aarch64-tests). They are NOT added to the VM test
set (macosx1500-aarch64-vms-tests) per the existing comment block at
that set: Bug 2038639 (GPU process leak in
APZCTreeManager::CheckerboardFlushObservers on headless macOS) blocks
web-platform-tests coverage on the VM platform. Once that bug
resolves, a follow-up can extend WPT sub-suite coverage to VMs.

This is the add-half of the cutover pattern. After validation,
a follow-up patch removes web-platform-tests-canvas and
web-platform-tests-webcodecs from macosx1470-64-tests.

These two WPT sub-suites were added to macosx1500-aarch64-tests in
D302779 (landed as dd16de76d667). Remove them from
macosx1470-64-tests now that the M4 platform is running them,
completing the WPT Batch 2 cutover.

Convention per recent migration commits (D294786 + D300150,
D301506 + D302482, D302779): each add-to-aarch64 patch should be
paired with a remove-from-Intel follow-up so Intel hardware load
actually drops.

After this lands, Intel (macosx1470-64-tests) retains:

  • cppunittest (intentionally cross-arch; stays)
  • crashtest (held pending Bug 2039098 fix)
  • firefox-ui-functional (single-suite eval pending)
  • gtest (deferred; needs arch-specific failure triage)
  • 3 web-platform-tests sub-suites still to be batched:
    eme (Widevine probe needed), print-reftest, reftest
    (both need fuzzy annotations for Apple Silicon)

firefox-ui-functional is a Marionette-driven UI automation suite that
currently runs on Linux, Windows, and Intel macOS (macosx1470-64-tests)
but not on the Sequoia M4 platform.

Adding it to macosx1500-aarch64-tests as a single-suite migration —
no batching needed since it's a standalone suite. After this lands
and stays green on autoland, a paired follow-up patch will drop it
from macosx1470-64-tests to complete the cutover.

This continues the pattern established by D294786 + D300150,
D301506 + D302482, D302779 + D303416.

The remaining Intel suites after this completes:

  • cppunittest (intentionally cross-arch; stays)
  • crashtest (held pending Bug 2039098)
  • gtest (held pending arch-specific failure triage)
  • 3 web-platform-tests sub-suites (eme, print-reftest, reftest)

firefox-ui-functional was added to macosx1500-aarch64-tests in D303794
(landed as 78b1268a4fc4 on autoland 2026-06-01). Remove it from
macosx1470-64-tests now that the M4 platform is running it,
completing the firefox-ui-functional cutover.

Convention per recent migration commits (D294786 + D300150,
D301506 + D302482, D302779 + D303416, D303794): each add-to-aarch64
patch should be paired with a remove-from-Intel follow-up so Intel
hardware load actually drops.

After this lands, Intel (macosx1470-64-tests) retains:

  • cppunittest (intentionally cross-arch; stays)
  • crashtest (held pending Bug 2039098 fix)
  • gtest (held; needs arch-specific failure triage)
  • 3 web-platform-tests sub-suites: eme, print-reftest, reftest

WPT Batch 4 — same pattern as D301506 (crashtest + wdspec),
D302779 (canvas + webcodecs), and the non-WPT cutovers.

Validation try push https://treeherder.mozilla.org/jobs?repo=try&revision=9cc569e29c79f588b1eda5e42266ec160ae9bcd4
returned 12/12 green, with both opt + debug eme tasks completing
cleanly on macosx1500-aarch64.

A comparison of WPT eme results between Intel macosx1470-64 and
M4 macosx1500-aarch64 confirms equivalent effective coverage:

Intel macosx1470-64 opt-eme: 43 clearkey OK, 50 drm SKIP
M4 macosx1500-aarch64 opt-eme: 48 clearkey OK, 47 drm SKIP + 4 TIMEOUT

Both platforms run clearkey tests (W3C standard, no real CDM
required) and skip the drm-* tests (Widevine-dependent). The 4
M4-specific TIMEOUTs match documented WPT metadata expectations
for macOS aarch64. CI has not had functional Widevine on macOS in
the recent past; this migration preserves the existing
(clearkey-only) coverage profile while moving load off Intel.

After this lands, a paired follow-up patch will remove
web-platform-tests-eme from macosx1470-64-tests to complete the
WPT Batch 4 cutover.

Note: if product wants functional Widevine CDM coverage on macOS
CI, that requires a separate CDM provisioning workstream and is
not addressed by or blocked by this migration.

web-platform-tests-eme was added to macosx1500-aarch64-tests in D303892
(landed as 243725382df0). Remove it from macosx1470-64-tests now that
the M4 platform is running it, completing the WPT Batch 4 cutover.

Convention per recent migration commits (D294786 + D300150,
D301506 + D302482, D302779 + D303416, D303794 + D303843, D303892):
each add-to-aarch64 patch should be paired with a remove-from-Intel
follow-up so Intel hardware load actually drops.

After this lands, Intel (macosx1470-64-tests) retains:

  • cppunittest (intentionally cross-arch; stays)
  • crashtest (held pending Bug 2039098 fix)
  • gtest (held; needs arch-specific failure triage)
  • 2 web-platform-tests sub-suites: print-reftest, reftest
    (WPT Batch 3, will need Apple Silicon fuzzy annotations)

WPT Batch 3 — same pattern as D301506 (crashtest + wdspec),
D302779 (canvas + webcodecs), D303892 (eme), and the earlier
non-WPT cutovers.

Validation try push https://treeherder.mozilla.org/jobs?repo=try&revision=fe1eeae8f78a7481e7b58f61ee616f5cef8d3f7d
(reftest add to M4) surfaced 2 stale meta annotations. Re-run after
cleanup (https://treeherder.mozilla.org/jobs?repo=try&revision=6431e1ff81a5a90fd595d9f5610b366d9e000b28)
showed 18/19 green with one remaining flake. This patch combines the
test-set add with the meta annotation cleanup:

  • testing/web-platform/meta/css/CSS2/backgrounds/background-root-016.xht.ini
    Single conditional expected: FAIL for OS X 15.3 was stale -- test now
    reliably passes on M4. Removed the entire meta file since no other
    expectations were authored.

  • testing/web-platform/meta/css/css-image-animation/image-animation-paused-exact-frame.tentative.html.ini
    Test is flaky on M4 (intermittent PASS/FAIL) -- same pattern as the
    existing OS X 14.7.5 annotation. Updated the OS X 15.3 line from
    bare FAIL to [PASS, FAIL] so both outcomes are documented as
    acceptable, matching the cross-version macOS opt-non-debug pattern.

Other meta annotations (e.g. fuzzy tolerances for Apple Silicon pixel
variance) were already in place from prior batches and didn't surface
additional issues during validation.

After this lands, a paired follow-up patch removes
web-platform-tests-reftest and web-platform-tests-print-reftest from
macosx1470-64-tests to complete the WPT Batch 3 cutover.

Pushed by rcurran@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/a11676d576e4 https://hg.mozilla.org/integration/autoland/rev/d6f210a2912f Add web-platform-tests-reftest and -print-reftest to macosx1500-aarch64-tests, clean up stale macOS 15.3 reftest expectations r=aryx

crashtest is already running on macosx1500-aarch64 (M4), added via
D294786 (Bug 2032601). M4 crashtest stability was confirmed in
Bug 2039098 (post-PPPC ~96% success across 1080 jobs on autoland).

This is the paired Intel-removal cutover; coverage is unchanged
(crashtest continues on M4). Frees Intel (Sonoma) hardware.

web-platform-tests-reftest and -print-reftest are intentionally NOT
dropped here: a validation try push surfaced 2 un-annotated reftest
failures on M4 (css-backgrounds/border-image-shorthand-001 and
forced-colors-mode-57) that need to be confirmed-intermittent and
annotated (or fixed) before removing the Intel signal. Their drop
will follow once M4 reftest is clean.

Depends on: 2045012
Pushed by abutkovits@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/a33de0d91527 https://hg.mozilla.org/integration/autoland/rev/17b205f65940 Revert "Bug 2034604 - Drop crashtest from macosx1470-64-tests r=aryx" for causing failures at pause-move-within-document.html.

web-platform-tests-reftest was added to macosx1500-aarch64 (M4) in
D304176. Three reftests need annotations for M4 that the Intel mac
annotations didn't cover:

  • css-backgrounds/border-image-shorthand-001.htm: Intel mac fuzz
    required exactly 32 differing pixels (totalPixels=32-32); M4 renders
    24 (max channel diff 1), below the minimum, so it failed. Widen mac
    range to totalPixels=0-32 to cover both. Sub-pixel AA only.

  • forced-colors-mode/forced-colors-mode-57.html: intermittent on M4 -
    observed both a clean PASS and a large rendering divergence (2015
    pixels, max channel diff 255) across try runs. Mark [PASS, FAIL] on
    os_version 15.30 (Intel mac stays PASS). Intermittent high-contrast
    rendering difference tracked in Bug 2045012.

  • css/CSS2/backgrounds/background-root-016.xht: intermittent sub-pixel
    AA difference on M4 (15 pixels, max channel diff 1); no prior
    annotation. Add a mac/15.30 fuzzy (maxDifference=0-1;totalPixels=0-20).

These are the blockers to dropping web-platform-tests-reftest and
-print-reftest from macosx1470-64-tests (Intel).

Attachment #9593556 - Attachment description: Bug 2034604 - Drop crashtest from macosx1470-64-tests r?aryx → Bug 2034604 - Drop crashtest from macosx1470-64-tests r=aryx
Attachment #9593716 - Attachment description: Bug 2034604 - Fix wpt-reftest annotations for macOS 15.30 AArch64 (M4) r?aryx → Bug 2034604 - Mark forced-colors-mode-57.html [PASS, FAIL] on macOS 15.30 AArch64 (M4) r=aryx

Final WPT cutover. Both suites are running on macosx1500-aarch64 (M4),
added via D304176 (WPT Batch 3). The M4-specific reftest annotations
they needed landed in D304867 (border-image-shorthand-001 fuzzy,
forced-colors-mode-57 [PASS,FAIL] on 15.30, background-root-016 fuzzy),
so M4 reftest is green.

This is the paired Intel-removal; coverage is unchanged (the suites
continue on M4). After this lands, macosx1470-64-tests is down to
cppunittest (kept on both platforms by design) and gtest (held pending
triage of arch-specific failures), completing the Intel->M4 test
migration except for gtest.

Pushed by rcurran@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/66e3417fec73 https://hg.mozilla.org/integration/autoland/rev/5dd640a10369 Drop web-platform-tests-reftest and -print-reftest from macosx1470-64-tests r=aryx
Depends on: 2045883
Flags: needinfo?(rcurran)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: