run macOS mochitests on AArch64
Categories
(Firefox Build System :: Task Configuration, task)
Tracking
(firefox152 fixed)
| Tracking | Status | |
|---|---|---|
| firefox152 | --- | fixed |
People
(Reporter: rcurran, Assigned: rcurran)
References
(Depends on 3 open bugs)
Details
Attachments
(20 files)
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review | |
|
48 bytes,
text/x-phabricator-request
|
Details | Review |
To reduce the load on the macOS Intel CPU test machines, mochitests should be moved to the macOS AArch64 workers
| Assignee | ||
Comment 1•3 months ago
|
||
| Assignee | ||
Comment 2•3 months ago
•
|
||
Move mochitest suites from macosx1470-64-tests (Intel) to
macosx1500-aarch64-tests (Apple Silicon M4).
Skip-if annotations added for known ARM failures (Bug 2034604):
- devtools/client/netmonitor browser_net_frame.js
- devtools/server/tests browser_canvasframe_helper_04.js
- devtools/client/framework browser_toolbox_meatball.js
- devtools/client/inspector/rules browser_rules_variables-jump-to-definition.js
- dom/events test_coalesce_touchmove.html, test_moz_mouse_pixel_scroll_event.html,
test_wheel_zoom_on_form_controls.html - dom/canvas/test/webgl-conf tex-unpack-params.html (debug crash)
- dom/credentialmanagement test_account_label.html
- dom/security/test/mixedcontentblocker test_bug803225.html
- dom/tests/mochitest/pointerlock test_pointerlock_xorigin_iframe_movementXY.html
- gfx/layers/apz test_group_touchevents-5.html
- accessible/tests/browser/tree browser_select.js
- image/test/mochitest test_animation_operators.html
- layout/style/test/chrome test_display_mode_reflow.html
- toolkit/components/aboutmemory test_aboutmemory5.xhtml, test_aboutmemory7.xhtml
- toolkit/content/tests/chrome test_menulist.xhtml
- toolkit/content/tests/widgets test_image_recognition_zh.html
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
https://treeherder.mozilla.org/jobs?repo=try&revision=b89a6f3ad24334d891229f0982c7f5ee9ce6958c
| Assignee | ||
Updated•3 months ago
|
Comment 4•3 months ago
|
||
No tasks on such platforms exist.
Comment 6•3 months ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/f23e041a36b1
https://hg.mozilla.org/mozilla-central/rev/fc9244e0a7a0
https://hg.mozilla.org/mozilla-central/rev/fa3bc3e61af1
Updated•3 months ago
|
Comment 9•3 months ago
|
||
A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)
Comment 10•3 months ago
|
||
The test fails permanently if only the manifest which contains it runs on
macOS 15.30 AArch64 debug.
Updated•3 months ago
|
Comment 11•3 months ago
|
||
Comment 12•3 months ago
|
||
| bugherder | ||
Comment 13•3 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 14•3 months ago
|
||
Move six suites off Intel onto the M4 fleet: jsreftest, reftest,
marionette-integration, marionette-unittest, telemetry-tests-client,
test-verify-wpt. All validated clean on macosx1500-aarch64/{debug,opt}
across two validation pushes after the two fuzzy annotations below.
Reftest fuzzy annotations for Apple Silicon-specific pixel variance:
- layout/reftests/box-shadow/boxshadow-rotated.html: M4 saw 5442
differing pixels (max 13), one above the existing 5250 ceiling.
Add fuzzy-if(appleSilicon,0-13,0-6000). - layout/reftests/bugs/1415987-1.html: M4 saw 6029 differing pixels
(max 255), four above the existing 6025 ceiling. Existing entry
already has a wider gtkWidget allowance; add a parallel
fuzzy-if(appleSilicon,0-255,0-6500).
Both are software-rendering pixel-rasterization differences across
architectures, not regressions.
Initially included gtest, test-verify, and test-verify-gpu but dropped:
- gtest hits arch-specific failures (PHC.TestPHCAllocations,
Jemalloc.BaseAllocMergeSplit, MediaDataEncoderTest.H264Encode4K*,
VP8VideoTrackEncoder.*, TestUDPSocket.TestUDPSocketMain). The PHC
test has a pending fix on Bug 2021381; the encoder/jemalloc tests
need their own evaluation. Moving gtest separately. - test-verify (debug+opt) fails in mozharness setup with
ModuleNotFoundError: No module named 'yaml' coming out of
manifestparser/toml.py. PyYAML is missing from the worker Python
env on gecko-t-osx-1500-m4 in the verify-tools path. Doesn't affect
test-verify-wpt, which uses a different mozharness path.
test-verify-gpu likely has the same issue but wasn't scheduled in
the validation push (no GPU test mods); deferring with test-verify.
crashtest is left alone here; it's covered by D294786 (Bug 2032601)
already in flight.
| Assignee | ||
Comment 15•3 months ago
|
||
These were initially included in D299130 but dropped after the validation
try push surfaced a missing pyyaml in the M4 worker Python env, causing
mozharness setup to fail with ModuleNotFoundError: No module named 'yaml'
during find_tests_for_verification. test-verify-wpt was unaffected
(different mozharness path) and migrated as part of D299130.
ronin_puppet PR #1204 (mozilla-platform-ops/ronin_puppet#1204) added
packages::python3_yaml to the gecko-t-osx-1500-m4 role; merged 2026-05-07
and propagating across the M4 fleet via per-task reboots. Confirmed
working on macmini-m4-111: pip3 show pyyaml reports 6.0.2 post-Puppet.
With pyyaml in place, test-verify can now run on M4. Move it and
test-verify-gpu off macosx1470-64-tests onto macosx1500-aarch64-tests.
Comment 16•3 months ago
|
||
Comment 17•3 months ago
|
||
| bugherder | ||
Comment 18•3 months ago
|
||
Comment 19•3 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 20•2 months ago
|
||
jittest and xpcshell were added to macosx1500-aarch64-tests in D294786
(landed as dc3288403045 on autoland 2026-05-12). Remove them from
macosx1470-64-tests now that M4 is running them, completing the cutover
for those two suites and dropping their duplicated load from the Intel
fleet.
Convention per recent migration commits (D299130, D299163): each
add-to-aarch64 patch should be paired with a remove-from-Intel
follow-up so Intel hardware load actually drops.
crashtest is intentionally left on Intel for now. Validation try push
showed pre-existing crashtest failures on macosx1500-aarch64 that
predate this patch (autoland has been ~80% fail on debug-crashtest and
~94% fail on opt-crashtest since D294786 landed). The failing tests
include dom/media/webrtc/tests/crashtests/2038533.html and
editor/libeditor/crashtests/1429523.{html,xhtml}, 1444630.html, all
with the same reftest-wait timeout pattern. Keeping crashtest on Intel
preserves a stable signal until those failures are resolved
separately.
After this lands, Intel (macosx1470-64-tests) retains:
- cppunittest (intentionally cross-arch; stays)
- crashtest (held pending M4 stability)
- firefox-ui-functional (single-suite eval pending)
- gtest (deferred; needs arch-specific failure triage)
- 7 web-platform-tests sub-suites (separate batch)
Comment 21•2 months ago
|
||
Comment 22•2 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 23•2 months ago
|
||
These are the two lowest-risk WPT sub-suites for the Sonoma Intel ->
Sequoia M4 migration: wdspec is WebDriver-protocol-driven with no
rendering dependency, and crashtest is "page loaded without crashing"
assertions with minimal rendering reliance.
This patch adds them to the hardware M4 test platform only
(macosx1500-aarch64-tests). They are intentionally not added to the
VM test set (macosx1500-aarch64-vms-tests) per the existing comment
block at that set: Bug 2038639 (GPU process leak in
APZCTreeManager::CheckerboardFlushObservers on headless macOS) blocks
web-platform-tests coverage on the VM platform. Once that bug
resolves, a follow-up can extend wdspec + crashtest to VMs.
This is the add-half of the cutover pattern established by D294786 /
D300150: add to M4, validate, then a follow-up patch removes the
sub-suites from macosx1470-64-tests.
Comment 24•2 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 25•2 months ago
|
||
These two WPT sub-suites were added to macosx1500-aarch64-tests in
D301506 (landed as 7a1a0928aceb on autoland 2026-05-20). Remove them
from macosx1470-64-tests now that the M4 platform is running them,
completing the cutover for WPT Batch 1 and dropping the duplicated
load from the Intel fleet.
Convention per recent migration commits (D294786 + D300150, D301506):
each add-to-aarch64 patch should be paired with a remove-from-Intel
follow-up so Intel hardware load actually drops.
After this lands, Intel (macosx1470-64-tests) retains:
- cppunittest (intentionally cross-arch; stays)
- crashtest (held pending Bug 2039098 fix)
- firefox-ui-functional (single-suite eval pending)
- gtest (deferred; needs arch-specific failure triage)
- 5 web-platform-tests sub-suites still to be batched:
canvas, eme, print-reftest, reftest, webcodecs
Comment 26•2 months ago
|
||
Comment 27•2 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 28•2 months ago
|
||
WPT Batch 2 — same pattern as D301506 (web-platform-tests-crashtest +
-wdspec) and the earlier non-WPT cutovers (D294786 + D300150).
canvas tests exercise the 2D rendering API; software-rasterizable in
practice, no GPU dependency required. webcodecs covers codec encode
and decode; the CPU codec path is used by default, hardware codec
behavior is not asserted by the WPT suite.
Both sub-suites are intentionally added to the hardware M4 test set
only (macosx1500-aarch64-tests). They are NOT added to the VM test
set (macosx1500-aarch64-vms-tests) per the existing comment block at
that set: Bug 2038639 (GPU process leak in
APZCTreeManager::CheckerboardFlushObservers on headless macOS) blocks
web-platform-tests coverage on the VM platform. Once that bug
resolves, a follow-up can extend WPT sub-suite coverage to VMs.
This is the add-half of the cutover pattern. After validation,
a follow-up patch removes web-platform-tests-canvas and
web-platform-tests-webcodecs from macosx1470-64-tests.
Comment 29•2 months ago
|
||
Comment 30•2 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 31•2 months ago
|
||
These two WPT sub-suites were added to macosx1500-aarch64-tests in
D302779 (landed as dd16de76d667). Remove them from
macosx1470-64-tests now that the M4 platform is running them,
completing the WPT Batch 2 cutover.
Convention per recent migration commits (D294786 + D300150,
D301506 + D302482, D302779): each add-to-aarch64 patch should be
paired with a remove-from-Intel follow-up so Intel hardware load
actually drops.
After this lands, Intel (macosx1470-64-tests) retains:
- cppunittest (intentionally cross-arch; stays)
- crashtest (held pending Bug 2039098 fix)
- firefox-ui-functional (single-suite eval pending)
- gtest (deferred; needs arch-specific failure triage)
- 3 web-platform-tests sub-suites still to be batched:
eme (Widevine probe needed), print-reftest, reftest
(both need fuzzy annotations for Apple Silicon)
Comment 32•2 months ago
|
||
Comment 33•2 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 34•2 months ago
|
||
firefox-ui-functional is a Marionette-driven UI automation suite that
currently runs on Linux, Windows, and Intel macOS (macosx1470-64-tests)
but not on the Sequoia M4 platform.
Adding it to macosx1500-aarch64-tests as a single-suite migration —
no batching needed since it's a standalone suite. After this lands
and stays green on autoland, a paired follow-up patch will drop it
from macosx1470-64-tests to complete the cutover.
This continues the pattern established by D294786 + D300150,
D301506 + D302482, D302779 + D303416.
The remaining Intel suites after this completes:
- cppunittest (intentionally cross-arch; stays)
- crashtest (held pending Bug 2039098)
- gtest (held pending arch-specific failure triage)
- 3 web-platform-tests sub-suites (eme, print-reftest, reftest)
Comment 35•2 months ago
|
||
| Assignee | ||
Comment 36•2 months ago
|
||
firefox-ui-functional was added to macosx1500-aarch64-tests in D303794
(landed as 78b1268a4fc4 on autoland 2026-06-01). Remove it from
macosx1470-64-tests now that the M4 platform is running it,
completing the firefox-ui-functional cutover.
Convention per recent migration commits (D294786 + D300150,
D301506 + D302482, D302779 + D303416, D303794): each add-to-aarch64
patch should be paired with a remove-from-Intel follow-up so Intel
hardware load actually drops.
After this lands, Intel (macosx1470-64-tests) retains:
- cppunittest (intentionally cross-arch; stays)
- crashtest (held pending Bug 2039098 fix)
- gtest (held; needs arch-specific failure triage)
- 3 web-platform-tests sub-suites: eme, print-reftest, reftest
| Assignee | ||
Comment 37•2 months ago
|
||
WPT Batch 4 — same pattern as D301506 (crashtest + wdspec),
D302779 (canvas + webcodecs), and the non-WPT cutovers.
Validation try push https://treeherder.mozilla.org/jobs?repo=try&revision=9cc569e29c79f588b1eda5e42266ec160ae9bcd4
returned 12/12 green, with both opt + debug eme tasks completing
cleanly on macosx1500-aarch64.
A comparison of WPT eme results between Intel macosx1470-64 and
M4 macosx1500-aarch64 confirms equivalent effective coverage:
Intel macosx1470-64 opt-eme: 43 clearkey OK, 50 drm SKIP
M4 macosx1500-aarch64 opt-eme: 48 clearkey OK, 47 drm SKIP + 4 TIMEOUT
Both platforms run clearkey tests (W3C standard, no real CDM
required) and skip the drm-* tests (Widevine-dependent). The 4
M4-specific TIMEOUTs match documented WPT metadata expectations
for macOS aarch64. CI has not had functional Widevine on macOS in
the recent past; this migration preserves the existing
(clearkey-only) coverage profile while moving load off Intel.
After this lands, a paired follow-up patch will remove
web-platform-tests-eme from macosx1470-64-tests to complete the
WPT Batch 4 cutover.
Note: if product wants functional Widevine CDM coverage on macOS
CI, that requires a separate CDM provisioning workstream and is
not addressed by or blocked by this migration.
Comment 38•2 months ago
|
||
Comment 39•2 months ago
|
||
Comment 40•2 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 41•2 months ago
|
||
web-platform-tests-eme was added to macosx1500-aarch64-tests in D303892
(landed as 243725382df0). Remove it from macosx1470-64-tests now that
the M4 platform is running it, completing the WPT Batch 4 cutover.
Convention per recent migration commits (D294786 + D300150,
D301506 + D302482, D302779 + D303416, D303794 + D303843, D303892):
each add-to-aarch64 patch should be paired with a remove-from-Intel
follow-up so Intel hardware load actually drops.
After this lands, Intel (macosx1470-64-tests) retains:
- cppunittest (intentionally cross-arch; stays)
- crashtest (held pending Bug 2039098 fix)
- gtest (held; needs arch-specific failure triage)
- 2 web-platform-tests sub-suites: print-reftest, reftest
(WPT Batch 3, will need Apple Silicon fuzzy annotations)
Comment 42•2 months ago
|
||
| Assignee | ||
Comment 43•2 months ago
|
||
WPT Batch 3 — same pattern as D301506 (crashtest + wdspec),
D302779 (canvas + webcodecs), D303892 (eme), and the earlier
non-WPT cutovers.
Validation try push https://treeherder.mozilla.org/jobs?repo=try&revision=fe1eeae8f78a7481e7b58f61ee616f5cef8d3f7d
(reftest add to M4) surfaced 2 stale meta annotations. Re-run after
cleanup (https://treeherder.mozilla.org/jobs?repo=try&revision=6431e1ff81a5a90fd595d9f5610b366d9e000b28)
showed 18/19 green with one remaining flake. This patch combines the
test-set add with the meta annotation cleanup:
-
testing/web-platform/meta/css/CSS2/backgrounds/background-root-016.xht.ini
Single conditionalexpected: FAILfor OS X 15.3 was stale -- test now
reliably passes on M4. Removed the entire meta file since no other
expectations were authored. -
testing/web-platform/meta/css/css-image-animation/image-animation-paused-exact-frame.tentative.html.ini
Test is flaky on M4 (intermittent PASS/FAIL) -- same pattern as the
existing OS X 14.7.5 annotation. Updated theOS X 15.3line from
bareFAILto[PASS, FAIL]so both outcomes are documented as
acceptable, matching the cross-version macOS opt-non-debug pattern.
Other meta annotations (e.g. fuzzy tolerances for Apple Silicon pixel
variance) were already in place from prior batches and didn't surface
additional issues during validation.
After this lands, a paired follow-up patch removes
web-platform-tests-reftest and web-platform-tests-print-reftest from
macosx1470-64-tests to complete the WPT Batch 3 cutover.
Comment 44•2 months ago
|
||
| bugherder | ||
Comment 45•2 months ago
|
||
Comment 46•2 months ago
|
||
| bugherder | ||
| Assignee | ||
Comment 47•2 months ago
|
||
crashtest is already running on macosx1500-aarch64 (M4), added via
D294786 (Bug 2032601). M4 crashtest stability was confirmed in
Bug 2039098 (post-PPPC ~96% success across 1080 jobs on autoland).
This is the paired Intel-removal cutover; coverage is unchanged
(crashtest continues on M4). Frees Intel (Sonoma) hardware.
web-platform-tests-reftest and -print-reftest are intentionally NOT
dropped here: a validation try push surfaced 2 un-annotated reftest
failures on M4 (css-backgrounds/border-image-shorthand-001 and
forced-colors-mode-57) that need to be confirmed-intermittent and
annotated (or fixed) before removing the Intel signal. Their drop
will follow once M4 reftest is clean.
Comment 48•2 months ago
|
||
Comment 49•2 months ago
|
||
Comment 50•2 months ago
|
||
Backed out for causing failures at pause-move-within-document.html.
Backout link: https://hg.mozilla.org/integration/autoland/rev/17b205f65940
Failure log: https://treeherder.mozilla.org/logviewer?job_id=570769361&repo=autoland&task=TiBlapciSrixZKe8YNefyQ.0&lineNumber=2145
| Assignee | ||
Comment 51•2 months ago
|
||
web-platform-tests-reftest was added to macosx1500-aarch64 (M4) in
D304176. Three reftests need annotations for M4 that the Intel mac
annotations didn't cover:
-
css-backgrounds/border-image-shorthand-001.htm: Intel mac fuzz
required exactly 32 differing pixels (totalPixels=32-32); M4 renders
24 (max channel diff 1), below the minimum, so it failed. Widen mac
range to totalPixels=0-32 to cover both. Sub-pixel AA only. -
forced-colors-mode/forced-colors-mode-57.html: intermittent on M4 -
observed both a clean PASS and a large rendering divergence (2015
pixels, max channel diff 255) across try runs. Mark [PASS, FAIL] on
os_version 15.30 (Intel mac stays PASS). Intermittent high-contrast
rendering difference tracked in Bug 2045012. -
css/CSS2/backgrounds/background-root-016.xht: intermittent sub-pixel
AA difference on M4 (15 pixels, max channel diff 1); no prior
annotation. Add a mac/15.30 fuzzy (maxDifference=0-1;totalPixels=0-20).
These are the blockers to dropping web-platform-tests-reftest and
-print-reftest from macosx1470-64-tests (Intel).
Updated•2 months ago
|
Comment 52•2 months ago
|
||
Updated•2 months ago
|
Comment 53•2 months ago
|
||
| Assignee | ||
Comment 54•2 months ago
|
||
Final WPT cutover. Both suites are running on macosx1500-aarch64 (M4),
added via D304176 (WPT Batch 3). The M4-specific reftest annotations
they needed landed in D304867 (border-image-shorthand-001 fuzzy,
forced-colors-mode-57 [PASS,FAIL] on 15.30, background-root-016 fuzzy),
so M4 reftest is green.
This is the paired Intel-removal; coverage is unchanged (the suites
continue on M4). After this lands, macosx1470-64-tests is down to
cppunittest (kept on both platforms by design) and gtest (held pending
triage of arch-specific failures), completing the Intel->M4 test
migration except for gtest.
Comment 55•2 months ago
|
||
Comment 56•2 months ago
|
||
| bugherder | ||
Comment 57•2 months ago
|
||
| bugherder | ||
| Assignee | ||
Updated•1 month ago
|
Description
•