mach bootstrap installs x86-64 binaries on arm64 windows

RESOLVED FIXED in Firefox 67

Status

P3
normal
RESOLVED FIXED
2 months ago
2 months ago

People

(Reporter: Gijs, Assigned: froydnj)

Tracking

(Blocks: 1 bug)

Trunk
mozilla67
ARM64
Windows 10

Firefox Tracking Flags

(firefox67 fixed)

Details

Attachments

(3 attachments)

(Reporter)

Description

2 months ago

+++ This bug was initially created as a clone of Bug #1486994 +++

mach bootstrap is useful for getting things set up, but it downloads x86-64 binaries of cbindgen, clang, and node.

(Also it fails to configure rust properly; it errors out. I have not dug deeply here but it looks like it downloads an x86-64 version of rustup-init and attempts to run that.)

Ideally we should provide arm64 binaries; where those are not available we should provide x86 (32-bit) binaries, which will at least run. We may want to warn in that case.

(Assignee)

Comment 1

2 months ago

(In reply to :Gijs (he/him) from comment #0)

+++ This bug was initially created as a clone of Bug #1486994 +++

mach bootstrap is useful for getting things set up, but it downloads x86-64 binaries of cbindgen, clang, and node.

(Also it fails to configure rust properly; it errors out. I have not dug deeply here but it looks like it downloads an x86-64 version of rustup-init and attempts to run that.)

Ideally we should provide arm64 binaries; where those are not available we should provide x86 (32-bit) binaries, which will at least run. We may want to warn in that case.

I think we should yell loudly if you're ever trying to install cbindgen and clang; I don't think we really want people doing full C++ builds on device...I'm not even sure that's supported.

Installing node, though, that should be very doable.

Assignee: nobody → nfroyd
Priority: -- → P3
(Assignee)

Comment 2

2 months ago
We don't use this in automation, but people who develop on AArch64
Windows will be happy we're providing Node for them.
Attachment #9041253 - Flags: review?(core-build-config-reviews)
(Assignee)

Comment 3

2 months ago
We don't use this yet, but it's easy enough to get out of the way.
Attachment #9041254 - Flags: review?(core-build-config-reviews)
(Assignee)

Comment 4

2 months ago
One less paper cut for frontend developers.
Attachment #9041255 - Flags: review?(core-build-config-reviews)
Comment on attachment 9041253 [details] [diff] [review]
part 1 - add a node repack task for 32-bit Windows

Review of attachment 9041253 [details] [diff] [review]:
-----------------------------------------------------------------

Fine by me.
Attachment #9041253 - Flags: review?(core-build-config-reviews) → review+
Comment on attachment 9041254 [details] [diff] [review]
part 2 - add a 32-bit node variable for bootstrapping

Review of attachment 9041254 [details] [diff] [review]:
-----------------------------------------------------------------

What about https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/windows.py#86?
Attachment #9041254 - Flags: review?(core-build-config-reviews) → review+
Comment on attachment 9041255 [details] [diff] [review]
part 3 - install 32-bit node on aarch64 devices

Review of attachment 9041255 [details] [diff] [review]:
-----------------------------------------------------------------

Technically, this looks fine.  Policy-wise, I don't understand

::: python/mozboot/mozboot/mozillabuild.py
@@ +90,5 @@
>          self.install_toolchain_artifact(state_dir, checkout_root, stylo.WINDOWS_CBINDGEN)
>  
>      def ensure_node_packages(self, state_dir, checkout_root):
>          from mozboot import node
> +        node_artifact = node.WIN32 if is_aarch64_host() else node.WIN64

This looks really surprising -- add a comment explaining why, please.
Attachment #9041255 - Flags: review?(core-build-config-reviews) → review+

I thought we had stated originally that we weren't going to support building on aarch64 hardware, only cross-compiling from x86-64 Windows? Has that changed?

(Assignee)

Comment 9

2 months ago

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #8)

I thought we had stated originally that we weren't going to support building on aarch64 hardware, only cross-compiling from x86-64 Windows? Has that changed?

Gijs is/was trying to do artifact builds directly on devices; front-endish type work seems reasonable to support, especially for speed of development.

Certainly the people attempting to stand up tests have resorted to installing old versions of MozillaBuild and whatnot to try and get all our testing programs running. So people are kind of doing this whether we want them to or not. I think it's OK to yell at people if they're trying to do full C++ builds, but everything else...might be OK?

Comment 10

2 months ago
Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/898be859c1a4
part 1 - add a node repack task for 32-bit Windows; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/ef7b4d60481f
part 2 - add a 32-bit node variable for bootstrapping; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/1a41624ff365
part 3 - install 32-bit node on aarch64 devices; r=nalexander

Backed out 3 changesets (Bug 1525069) for linting failure at /builds/worker/checkouts/gecko/python/mozboot/mozboot/mozillabuild.p on a CLOSED TREE

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=1a41624ff365f9ea3905ae765c32bc97934cfd3d

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=225990140&repo=mozilla-inbound&lineNumber=255

Backout link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=9936bc3bd7061b41a03306703ac078bc62a4cf28

[vcs 2019-02-04T21:39:14.895Z] TinderboxPrint:<a href=https://hg.mozilla.org/integration/mozilla-inbound/rev/1a41624ff365f9ea3905ae765c32bc97934cfd3d title='Built from mozilla-inbound revision 1a41624ff365f9ea3905ae765c32bc97934cfd3d'>1a41624ff365f9ea3905ae765c32bc97934cfd3d</a>
[task 2019-02-04T21:39:14.895Z] executing ['bash', '-cx', 'cd $GECKO_PATH && ./mach lint -l py2 -l py3 -f treeherder']
[task 2019-02-04T21:39:14.902Z] + cd /builds/worker/checkouts/gecko
[task 2019-02-04T21:39:14.902Z] + ./mach lint -l py2 -l py3 -f treeherder
[task 2019-02-04T21:39:15.676Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2019-02-04T21:39:15.676Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2019-02-04T21:39:17.251Z] Installing setuptools, pip, wheel...done.
[task 2019-02-04T21:39:18.314Z] running build_ext
[task 2019-02-04T21:39:18.314Z] building 'psutil._psutil_linux' extension
[task 2019-02-04T21:39:18.314Z] creating build
[task 2019-02-04T21:39:18.314Z] creating build/temp.linux-x86_64-2.7
[task 2019-02-04T21:39:18.314Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2019-02-04T21:39:18.314Z] creating build/lib.linux-x86_64-2.7
[task 2019-02-04T21:39:18.314Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2019-02-04T21:39:18.314Z] building 'psutil._psutil_posix' extension
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-02-04T21:39:18.315Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-02-04T21:39:18.315Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-02-04T21:39:18.315Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-02-04T21:39:18.315Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-02-04T21:39:18.315Z]
[task 2019-02-04T21:39:18.315Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-02-04T21:39:23.021Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozboot/mozboot/mozillabuild.py:35:37 | invalid syntax (is-parseable)
[task 2019-02-04T21:39:23.021Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozboot/mozboot/mozillabuild.py:35:37 | invalid syntax (is-parseable)
[taskcluster 2019-02-04 21:39:23.852Z] === Task Finished ===
[taskcluster 2019-02-04 21:39:23.852Z] Unsuccessful task run with exit code: 1 completed in 267.855 seconds

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #8)

I thought we had stated originally that we weren't going to support building on aarch64 hardware, only cross-compiling from x86-64 Windows?

I hope that is not true, since everytime I have to use Windows (for almost anything) I become very moody and start wondering if I should find another line of work. It's not a state of mind I enjoy.

Jokes aside: Certainly for working on the JS and Wasm JITs, being able to get a reasonable build environment set up on some ARM64 device so that it's possible to compile and debug and run JS shell tests (as many as possible) locally is a major productivity booster. The ARM64 simulator is no substitute at all. I know this is not the full browser, but the fewer hurdles for this work there is, the better.

(In reply to Lars T Hansen [:lth] from comment #12)

Jokes aside: Certainly for working on the JS and Wasm JITs, being able to
get a reasonable build environment set up on some ARM64 device so that it's
possible to compile and debug and run JS shell tests (as many as possible)
locally is a major productivity booster. The ARM64 simulator is no
substitute at all. I know this is not the full browser, but the fewer
hurdles for this work there is, the better.

FWIW I was able to copy an objdir over from my compile machine and run jit_test.py on my ARM64 laptop. I know it's not as frictionless as a native development environment but hopefully that's less bad than the simulator.

If the work to stand up a JS shell build is substantially less than a full browser build, I wouldn't be opposed to seeing it land, but I don't think anyone's currently signed up to do the work.

Comment 14

2 months ago
Pushed by nfroyd@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ce5026b27e78
part 1 - add a node repack task for 32-bit Windows; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/027ed877d832
part 2 - add a 32-bit node variable for bootstrapping; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7b9d2f169d5
part 3 - install 32-bit node on aarch64 devices; r=nalexander

Comment 15

2 months ago
bugherder
Status: NEW → RESOLVED
Last Resolved: 2 months ago
status-firefox67: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla67
You need to log in before you can comment on or make changes to this bug.