1690870 - Artifact builds don't work on M1 macs

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P3)

Description

[Brian Grinstead [:bgrins]]

mozconfig:

mk_add_options AUTOCLOBBER=1
mk_add_options MOZ_OBJDIR=objdir-opt-artifact.noindex
ac_add_options --enable-artifact-builds

As per Haik:

> It appears to be a codesigning problem because the executables have no codesigning.
$ codesign -dvvv /Users/haftandilian/r/mc/artifact.noindex/dist/Nightly.app/Contents/MacOS/firefox
/Users/haftandilian/r/mc/artifact.noindex/dist/Nightly.app/Contents/MacOS/firefox: code object is not signed at all

> The M1 requires everything be codesigned in order to execute. This happens automatically for things you build locally.

> This is separate from the bundle signing we explicitly do for release builds.

Comment 1

[Brian Grinstead [:bgrins]]

Here's a workaround:

#!/bin/bash
set -x
find $1 -type f -exec codesign -s - {} \;
export TMP_FIRERFOX_BIN=`mktemp`
cp ${ART_OBJ_DIR}/dist/Nightly.app/Contents/MacOS/firefox ${TMP_FIREFOX_BIN}
codesign -s - ${TMP_FIREFOX_BIN}
cp ${TMP_FIREFOX_BIN} ${ART_OBJ_DIR}/dist/Nightly.app/Contents/MacOS/firefox
rm ${TMP_FIREFOX_BIN}

And then

./self-sign-artifact-objdir.bash ./artifact-objdir

Comment 2

[Mike Hommey [:glandium]]

export TMP_FIRERFOX_BIN=mktemp

There's a typo here.

I guess we need to bite the bullet and implement signing for local builds... because we don't have a signed version of the non-unified builds. We do have the signature script in tree, but it would probably need some adjustment to work locally without too much hassle.

Comment 3

[Haik Aftandilian [:haik]]

I've been working on an effort to integrate my codesign-tree[1] python script into iscript to make our codesigning controlled by a json input file that resides in the tree. We could use that here once we determine why codesign is aborting on some of the files in the artifact dist/Nightly.app bundle. See example below that codesigns the artifact Nightly.app bundle.

Note: signing with codesign-tree or re-using code we have on infra to sign local builds like we do on CI would be an improvement because we would be applying our entitlements and enabling hardened runtime restrictions which is not yet enabled on local builds. Bug 1522409 is filed for this.

The motivation for using codesign-tree is that in the future we'll be using different entitlement files for the parent and child process executables, and different executables for different child process types, and this allows us to control the codesigning arguments from a file that lives in the tree. Developers wanting to sign with Apple codesign identities would just need to override the -s identity option.

$ git clone https://github.com/hafta/codesign-tree.git codesign-tree
$ cd mozilla-central
$ python3 ../codesign-tree/codesigntree.py \
-s - \
-m ../codesign-tree/examples/01-firefox/codesign-map.json \
-d security/mac/hardenedruntime/ \
-r artifact.noindex/dist/Nightly.app

1. https://github.com/hafta/codesign-tree

Comment 4

[Harry Twyford]

fwiw, the front-end team is hiring several engineers soon. Assuming at least some of them decide to procure MacBooks, that will be a significant increase in people who are using M1 Macs and want to use artifact builds. Even if we can't fix this bug in time, we might want to add a message to mach and/or our Building Firefox On macOS docs mentioning this issue. I think that would mitigate some headaches.

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1690870: Remove "artifact mode" from bootstrap options on M1 Macs

As pointed out by Harry, this will help in the short-term with
onboarding new developers who will be using M1 macs.

Note that this patch doesn't leverage the "instance" classes to define
whether they're artifact-mode compatible or not, and that's because:

• Most (all except one?) of the systems support artifact mode, therefore
• Since this is a temporary workaround, it's more deletable to have
  a top-level if/else than to add a supports_artifact_mode property
  to each Bootstrapper.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ca89299f8203
Remove "artifact mode" from bootstrap options on M1 Macs r=firefox-build-system-reviewers,andi

Comment 7

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/ca89299f8203

Comment 8

[Gavin Suntop [:gvn] (they/them)]

Has there been any further progress made to fix this issue? If not is there any way this can receive higher priority?

All current generation MacBook Pros are now using M1 chips. I haven't been able to get the above workarounds to succeed.

Thank you!

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Hmm, good question, I don't have a strong handle on the current state of this.
I'll redirect over to glandium, who will have a better idea.

Comment 10

[Mike Hommey [:glandium]]

I had this in my pipe until clang 13 and rust 1.56 problems took me away from it. I've now resumed working on this, and will have patches shortly.

Comment 11

[Mike Hommey [:glandium]]

Attachment: Bug 1690870 - Stop getting artifacts for host binaries.

Ever since all builds we get artifacts from have been cross-compiled, those
artifacts have been for Linux. They are not of any use on Mac or Windows hosts,
so it's unnecessary to get them. And we might as well be consistent across
platforms, and not get them on Linux either, even if they are native.

Comment 12

[Mike Hommey [:glandium]]

Attachment: Bug 1690870 - Re-sign Mach-O binaries on macOS.

Comment 13

[Mike Hommey [:glandium]]

Attachment: Bug 1690870 - Enable bootstrap for artifact builds on arm64 macs.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/05556177ebe6
Stop getting artifacts for host binaries. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/eb0be4532ace
Re-sign Mach-O binaries on macOS. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/258ae2eb6cd4
Enable bootstrap for artifact builds on arm64 macs. r=firefox-build-system-reviewers,andi

Comment 15

[Cristina Cozmuta (:CrissCozmuta)]

Backed out for causing build bustages. CLOSED TREE

Backout link : https://hg.mozilla.org/integration/autoland/rev/ef1432d50aa10653bccea1c089d8b5d292e0304b

Push with failures : https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Crunnable&revision=258ae2eb6cd4d09aa3885a8a3a741e47b65de7d4&selectedTaskRun=EPNT8J17SBuwMqOvtTeF7g.0

Link to failures log :
https://treeherder.mozilla.org/logviewer?job_id=356317605&repo=autoland&lineNumber=5439
https://treeherder.mozilla.org/logviewer?job_id=356317611&repo=autoland&lineNumber=36851
https://treeherder.mozilla.org/logviewer?job_id=356317738&repo=autoland&lineNumber=1685
https://treeherder.mozilla.org/logviewer?job_id=356317780&repo=autoland&lineNumber=7115

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/15bced359517
Stop getting artifacts for host binaries. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/d45fe40b6b22
Re-sign Mach-O binaries on macOS. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/e6db9f533117
Enable bootstrap for artifact builds on arm64 macs. r=firefox-build-system-reviewers,andi

Comment 17

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/15bced359517
https://hg.mozilla.org/mozilla-central/rev/d45fe40b6b22
https://hg.mozilla.org/mozilla-central/rev/e6db9f533117

Comment 18

[Gavin Suntop [:gvn] (they/them)]

This seems to be working now after re-bootstrapping. Thank you so much!