1525069 - mach bootstrap installs x86-64 binaries on arm64 windows

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, enhancement, P3)

Description

[:Gijs (he/him)]

+++ This bug was initially created as a clone of Bug #1486994 +++

mach bootstrap is useful for getting things set up, but it downloads x86-64 binaries of cbindgen, clang, and node.

(Also it fails to configure rust properly; it errors out. I have not dug deeply here but it looks like it downloads an x86-64 version of rustup-init and attempts to run that.)

Ideally we should provide arm64 binaries; where those are not available we should provide x86 (32-bit) binaries, which will at least run. We may want to warn in that case.

Comment 1

[Nathan Froyd [:froydnj]]

(In reply to :Gijs (he/him) from comment #0)

+++ This bug was initially created as a clone of Bug #1486994 +++

mach bootstrap is useful for getting things set up, but it downloads x86-64 binaries of cbindgen, clang, and node.

(Also it fails to configure rust properly; it errors out. I have not dug deeply here but it looks like it downloads an x86-64 version of rustup-init and attempts to run that.)

Ideally we should provide arm64 binaries; where those are not available we should provide x86 (32-bit) binaries, which will at least run. We may want to warn in that case.

I think we should yell loudly if you're ever trying to install cbindgen and clang; I don't think we really want people doing full C++ builds on device...I'm not even sure that's supported.

Installing node, though, that should be very doable.

Comment 2

[Nathan Froyd [:froydnj]]

Attachment: part 1 - add a node repack task for 32-bit Windows

We don't use this in automation, but people who develop on AArch64
Windows will be happy we're providing Node for them.

Comment 3

[Nathan Froyd [:froydnj]]

Attachment: part 2 - add a 32-bit node variable for bootstrapping

We don't use this yet, but it's easy enough to get out of the way.

Comment 4

[Nathan Froyd [:froydnj]]

Attachment: part 3 - install 32-bit node on aarch64 devices

One less paper cut for frontend developers.

Comment 5

[Nick Alexander :nalexander [he/him]]

Comment on attachment 9041253 [details] [diff] [review]
part 1 - add a node repack task for 32-bit Windows

Review of attachment 9041253 [details] [diff] [review]:
-----------------------------------------------------------------

Fine by me.

Comment 6

[Nick Alexander :nalexander [he/him]]

Comment on attachment 9041254 [details] [diff] [review]
part 2 - add a 32-bit node variable for bootstrapping

Review of attachment 9041254 [details] [diff] [review]:
-----------------------------------------------------------------

What about https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/windows.py#86?

Comment 7

[Nick Alexander :nalexander [he/him]]

Comment on attachment 9041255 [details] [diff] [review]
part 3 - install 32-bit node on aarch64 devices

Review of attachment 9041255 [details] [diff] [review]:
-----------------------------------------------------------------

Technically, this looks fine.  Policy-wise, I don't understand

::: python/mozboot/mozboot/mozillabuild.py
@@ +90,5 @@
>          self.install_toolchain_artifact(state_dir, checkout_root, stylo.WINDOWS_CBINDGEN)
>  
>      def ensure_node_packages(self, state_dir, checkout_root):
>          from mozboot import node
> +        node_artifact = node.WIN32 if is_aarch64_host() else node.WIN64

This looks really surprising -- add a comment explaining why, please.

Comment 8

[(not currently active) Ted Mielczarek]

I thought we had stated originally that we weren't going to support building on aarch64 hardware, only cross-compiling from x86-64 Windows? Has that changed?

Comment 9

[Nathan Froyd [:froydnj]]

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #8)

I thought we had stated originally that we weren't going to support building on aarch64 hardware, only cross-compiling from x86-64 Windows? Has that changed?

Gijs is/was trying to do artifact builds directly on devices; front-endish type work seems reasonable to support, especially for speed of development.

Certainly the people attempting to stand up tests have resorted to installing old versions of MozillaBuild and whatnot to try and get all our testing programs running. So people are kind of doing this whether we want them to or not. I think it's OK to yell at people if they're trying to do full C++ builds, but everything else...might be OK?

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/898be859c1a4
part 1 - add a node repack task for 32-bit Windows; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/ef7b4d60481f
part 2 - add a 32-bit node variable for bootstrapping; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/1a41624ff365
part 3 - install 32-bit node on aarch64 devices; r=nalexander

Comment 11

[Daniel Varga [:dvarga]]

Backed out 3 changesets (Bug 1525069) for linting failure at /builds/worker/checkouts/gecko/python/mozboot/mozboot/mozillabuild.p on a CLOSED TREE

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=1a41624ff365f9ea3905ae765c32bc97934cfd3d

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=225990140&repo=mozilla-inbound&lineNumber=255

Backout link: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&revision=9936bc3bd7061b41a03306703ac078bc62a4cf28

[vcs 2019-02-04T21:39:14.895Z] TinderboxPrint:<a href=https://hg.mozilla.org/integration/mozilla-inbound/rev/1a41624ff365f9ea3905ae765c32bc97934cfd3d title='Built from mozilla-inbound revision 1a41624ff365f9ea3905ae765c32bc97934cfd3d'>1a41624ff365f9ea3905ae765c32bc97934cfd3d</a>
[task 2019-02-04T21:39:14.895Z] executing ['bash', '-cx', 'cd $GECKO_PATH && ./mach lint -l py2 -l py3 -f treeherder']
[task 2019-02-04T21:39:14.902Z] + cd /builds/worker/checkouts/gecko
[task 2019-02-04T21:39:14.902Z] + ./mach lint -l py2 -l py3 -f treeherder
[task 2019-02-04T21:39:15.676Z] New python executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python2.7
[task 2019-02-04T21:39:15.676Z] Also creating executable in /builds/worker/checkouts/gecko/obj-x86_64-pc-linux-gnu/_virtualenvs/init/bin/python
[task 2019-02-04T21:39:17.251Z] Installing setuptools, pip, wheel...done.
[task 2019-02-04T21:39:18.314Z] running build_ext
[task 2019-02-04T21:39:18.314Z] building 'psutil._psutil_linux' extension
[task 2019-02-04T21:39:18.314Z] creating build
[task 2019-02-04T21:39:18.314Z] creating build/temp.linux-x86_64-2.7
[task 2019-02-04T21:39:18.314Z] creating build/temp.linux-x86_64-2.7/psutil
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o
[task 2019-02-04T21:39:18.314Z] creating build/lib.linux-x86_64-2.7
[task 2019-02-04T21:39:18.314Z] creating build/lib.linux-x86_64-2.7/psutil
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o build/temp.linux-x86_64-2.7/psutil/_psutil_linux.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so
[task 2019-02-04T21:39:18.314Z] building 'psutil._psutil_posix' extension
[task 2019-02-04T21:39:18.314Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_common.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_common.o
[task 2019-02-04T21:39:18.315Z] x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=543 -DPSUTIL_LINUX=1 -I/usr/include/python2.7 -c psutil/_psutil_posix.c -o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o
[task 2019-02-04T21:39:18.315Z] x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-Bsymbolic-functions -Wl,-z,relro -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/psutil/_psutil_common.o build/temp.linux-x86_64-2.7/psutil/_psutil_posix.o -o build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so
[task 2019-02-04T21:39:18.315Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_linux.so -> psutil
[task 2019-02-04T21:39:18.315Z] copying build/lib.linux-x86_64-2.7/psutil/_psutil_posix.so -> psutil
[task 2019-02-04T21:39:18.315Z]
[task 2019-02-04T21:39:18.315Z] Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
[task 2019-02-04T21:39:23.021Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozboot/mozboot/mozillabuild.py:35:37 | invalid syntax (is-parseable)
[task 2019-02-04T21:39:23.021Z] TEST-UNEXPECTED-ERROR | /builds/worker/checkouts/gecko/python/mozboot/mozboot/mozillabuild.py:35:37 | invalid syntax (is-parseable)
[taskcluster 2019-02-04 21:39:23.852Z] === Task Finished ===
[taskcluster 2019-02-04 21:39:23.852Z] Unsuccessful task run with exit code: 1 completed in 267.855 seconds

Comment 12

[Lars T Hansen [:lth]]

(In reply to Ted Mielczarek [:ted] [:ted.mielczarek] from comment #8)

I thought we had stated originally that we weren't going to support building on aarch64 hardware, only cross-compiling from x86-64 Windows?

I hope that is not true, since everytime I have to use Windows (for almost anything) I become very moody and start wondering if I should find another line of work. It's not a state of mind I enjoy.

Jokes aside: Certainly for working on the JS and Wasm JITs, being able to get a reasonable build environment set up on some ARM64 device so that it's possible to compile and debug and run JS shell tests (as many as possible) locally is a major productivity booster. The ARM64 simulator is no substitute at all. I know this is not the full browser, but the fewer hurdles for this work there is, the better.

Comment 13

[(Away)]

(In reply to Lars T Hansen [:lth] from comment #12)

Jokes aside: Certainly for working on the JS and Wasm JITs, being able to
get a reasonable build environment set up on some ARM64 device so that it's
possible to compile and debug and run JS shell tests (as many as possible)
locally is a major productivity booster. The ARM64 simulator is no
substitute at all. I know this is not the full browser, but the fewer
hurdles for this work there is, the better.

FWIW I was able to copy an objdir over from my compile machine and run jit_test.py on my ARM64 laptop. I know it's not as frictionless as a native development environment but hopefully that's less bad than the simulator.

If the work to stand up a JS shell build is substantially less than a full browser build, I wouldn't be opposed to seeing it land, but I don't think anyone's currently signed up to do the work.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ce5026b27e78
part 1 - add a node repack task for 32-bit Windows; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/027ed877d832
part 2 - add a 32-bit node variable for bootstrapping; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7b9d2f169d5
part 3 - install 32-bit node on aarch64 devices; r=nalexander

Comment 15

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/ce5026b27e78
https://hg.mozilla.org/mozilla-central/rev/027ed877d832
https://hg.mozilla.org/mozilla-central/rev/a7b9d2f169d5