2051353 - Fix and re-enable the mozscreenshots screenshot jobs on Linux, Windows, and macOS

Status: RESOLVED FIXED

(Testing :: mozscreenshots, task)

Description

[Florian Quèze [:florian]]

The mochitest-browser-screenshots jobs (the in-tree mozscreenshots suite under browser/tools/mozscreenshots/) do not currently produce usable screenshot artifacts:

• On the branches where they run (autoland, mozilla-central), capture is skipped entirely because shouldCapture() bails out on the nightly channel, so no images are ever taken or uploaded.
• The macOS capture path in Screenshot.sys.mjs (_screenshotOSX) relied on an AppleScript (osascript) window-id lookup that times out on CI workers without the macOS Automation permission, which is why the macOS configurations have been disabled for years (see bug 1554821).
• The jobs ran on Linux only.

This bug covers the engineering work to make these jobs run and upload screenshots on all three desktop platforms:

• Capture and upload whenever MOZ_UPLOAD_DIR is set, instead of skipping on the nightly channel.
• Make _screenshotOSX capture the screen directly with screencapture -x, dropping the osascript window-id lookup.
• Re-enable the macOS configurations that were disabled due to the Apple Event timeout.
• Run the jobs on Linux, Windows, and macOS.
• Add a per-platform Taskcluster index route to each job (under index.gecko.v2.<project>.latest.source.mochitest-browser-screenshots.<test-platform>) so a dashboard can fetch the latest captured artifacts for a given platform without knowing the task id.

Adapting the set of screenshots actually taken to match the UX team's shot list (bug 2050672) is out of scope here and will be handled in separate bugs.

Comment 1

[Florian Quèze [:florian]]

Attachment: Bug 2051353 - Capture and upload mozscreenshots screenshots whenever there is an upload directory. r=#firefox-desktop-core-reviewers

shouldCapture() previously bailed out on the nightly channel, which meant the
mochitest-browser-screenshots jobs running on autoland and mozilla-central never
captured anything. Gate on MOZ_UPLOAD_DIR instead, so screenshots are captured
in CI (where there is a directory to upload them to) and skipped on local builds.

Also upload to MOZ_UPLOAD_DIR unconditionally rather than excluding integration
branches, so the images are available as artifacts on every branch that runs.

Comment 2

[Florian Quèze [:florian]]

Attachment: Bug 2051353 - Fix mozscreenshots screen capture on macOS. r=#firefox-desktop-core-reviewers

_screenshotOSX() ran an osascript to look up the frontmost window's id before
capturing. Sending that Apple Event requires the macOS Automation permission,
which CI workers don't grant, so it timed out (AppleEvent timed out, -1712) and
the suite expanded to consume its whole timeout budget. This is the root cause
behind the long-standing macOS mozscreenshots skips (see bug 1554821).

The window id was never actually used by the capture (screencapture already grabs
the whole screen), so drop the osascript step and call screencapture -x directly,
matching what the Linux and Windows paths do.

Re-enable browser_primaryUI.js, browser_controlCenter.js and browser_preferences.js
on macosx1500-aarch64 now that this timeout is fixed.

browser_screenshots_cropping.js compares canvas-cropped images against reference
PNGs; on Apple Silicon the cropped output differs from the references by a small
number of edge pixels. Skip it on macosx1500-aarch64 for now; this is independent
of the capture mechanism and was never exercised while the suite was disabled.

Depends on D309408

Comment 3

[Florian Quèze [:florian]]

Attachment: Bug 2051353 - Add per-platform Taskcluster index routes to test tasks. r=#taskgraph-reviewers

Test tasks could not be indexed: the test description schema had no index field.
Add one (reusing the job description's IndexSchema) and thread it through to the
job description.

Because the same suite runs on several test platforms, append the test-platform
to the index job-name so the per-platform tasks get distinct index paths instead
of clobbering each other, and skip indexing the confirm-failure (-cf) copies,
which never run on real branches.

Give mochitest-browser-screenshots an index block (product source) so its
captured artifacts are discoverable at
index.gecko.v2.<project>.latest.source.mochitest-browser-screenshots.<test-platform>.

Depends on D309409

Comment 4

[Florian Quèze [:florian]]

Attachment: Bug 2051353 - Run the mozscreenshots jobs on Windows and macOS. r=#taskgraph-reviewers

Run mochitest-browser-screenshots on Windows (windows11-64-25h2) and macOS
(macosx1500-aarch64) in addition to Linux, and add the desktop-screenshot-capture
test set to macosx1500-aarch64/opt so the suite is scheduled there.

Drop the no-fission variant: Fission is the default and the extra variant adds no
useful coverage for these screenshots.

Depends on D309410

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/3cf0cca0b19a
https://hg.mozilla.org/integration/autoland/rev/d47fa8214ab0
Capture and upload mozscreenshots screenshots whenever there is an upload directory. r=firefox-desktop-core-reviewers ,Gijs
https://github.com/mozilla-firefox/firefox/commit/0d95ba0f3690
https://hg.mozilla.org/integration/autoland/rev/53bf26f90f4c
Fix mozscreenshots screen capture on macOS. r=firefox-desktop-core-reviewers ,Gijs
https://github.com/mozilla-firefox/firefox/commit/f036d006abb5
https://hg.mozilla.org/integration/autoland/rev/461ee7c29540
Add per-platform Taskcluster index routes to test tasks. r=taskgraph-reviewers,ahal
https://github.com/mozilla-firefox/firefox/commit/4b38a1e2e367
https://hg.mozilla.org/integration/autoland/rev/0a6564c0ad0a
Run the mozscreenshots jobs on Windows and macOS. r=taskgraph-reviewers,ahal

Comment 6

[amarc]

https://hg.mozilla.org/mozilla-central/rev/d47fa8214ab0
https://hg.mozilla.org/mozilla-central/rev/53bf26f90f4c
https://hg.mozilla.org/mozilla-central/rev/461ee7c29540
https://hg.mozilla.org/mozilla-central/rev/0a6564c0ad0a