Building Thunderbird 102 on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments June 16 PDT)
Categories
(Thunderbird :: Build Config, defect)
Tracking
(Not tracked)
People
(Reporter: daniel.bonner, Unassigned)
References
(Blocks 1 open bug)
Details
Attachments
(6 files)
User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0
Steps to reproduce:
Building 102.0a1 on aarch64 arm64 has wasm-ld errors
Summary:
I am running Amazon EC2 instance (ami-078d455b3b6ad0d4e, Ubuntu Jammy, aarch64) with the aim of building Thunderbird 91 and Thunderbird 102, both outputting aarch64 packages.
Note: Thunderbird 91 builds OK.
Note: Thunderbird 102 has the following wasm-ld errors during the "./mach build" process. When one of these errors is fixed, I could re-run the build process until the next wasm-ld error occurred:
-
wasm-ld: error: unable to find library -lgcc
-
wasm-ld: error: unknown file type: _muldi3.o
-
wasm-ld: error: unknown file type: letf2.o
-
wasm-ld: error: unknown file type: trunctfdf2.o
-
wasm-ld: error: undefined symbols (8 lines of errors each naming different symbols)
-
Finally ld throws an error from rlbox.wasm.c
I am able to fix the wasm-ld errors by:
A. Forcing wasm-ld to have these default flags:
-L(directory containing a custom libgcc.a)
--allow-undefined
B. Removing the unknown file types from my custom libgcc.a :
ar dv libgcc.a _muldi3.o
ar dv libgcc.a letf2.o
ar dv libgcc.a trunctfdf2.o
C. Deleting line 706306 in rlbox.wasm.c (although technically this is an ld not wasm-ld error)
After implementing fixes (A to C), the Thunderbird 102 build succeeds. The result from ./mach package works perfectly (tested on aarch64 devices).
I set up the system for new builds of Thunderbird 102 by keeping my custom wasm-ld binary selected with update-alternatives and making it use a custom libgcc.a
However, I need to wait until the Thunderbird 102 build produces the final error (rlbox.wasm.c:706306: undefined reference to `Z_envZ___multi3Z_vijjjj') before I can delete the offending line 706306. Finally, I rerun ./mach build to a successful build.
Detailed steps:
Note: I use llvm-13 and gcc 12.1 below. However, I have also tried llvm-10, llvm-13, llvm-14 and llvm-15 (nightly) combined with gcc-9.3, gcc-11.2 and gcc-12.1. The same errors occur in multiple combinations of gcc and llvm.
sudo apt update && sudo apt upgrade
sudo apt install mlocate git mercurial python3 python3-pip ansible nano watchman python3 python3-pip python3-setuptools python3-wheel default-jre default-jdk gcc g++ binutils libc6 libc6-dev libgcc-11-dev libstdc++-11-dev libstdc++6 linux-libc-dev libstdc++6 libstdc++-11-dev libx11-dev libxext-dev libxt-dev libxcb1-dev libxcb-shm0-dev libx11-xcb-dev clang-13 clang-tools-13 clang-format-13 clangd-13 clang-tidy-13 libclang-13-dev libclang-common-13-dev libclang-cpp13 libclang1-13 libclang-cpp13-dev llvm-13 llvm-13-runtime libllvm13 llvm-13-dev libc++1 libc++abi1 libc++-dev libgtk-3-dev libdbus-glib-1-dev wget curl git gcc g++ python3 python3-pip build-essential cmake ninja-build fonts-powerline powerline build-essential python3-pip cmake ninja-build python3 libc++1-13 libc++abi1-13 libunwind-13 lld-13 libbsd-dev emscripten wasi-libc libunwind-13 libunwind-13-dev libc++abi-13-dev libc++-13-dev gcc-12 g++-12 shc
Install Rust 1.61 and 1.53 with:
./rustup-init.sh -y --default-toolchain stable --default-host aarch64-unknown-linux-gnu --component rustfmt
Install cbindgen 0.23.0 with:
cargo install cbindgen
Install NodeJS 16.15.0 with nvm:
nvm install 16.15.0
nvm use 16.15.0
nvm alias default 16.15.0
Install npm 8.11.0 with:
npm install -g npm@latest
Contents of mozconfig:
ac_add_options --target=aarch64-unknown-linux-gnu
ac_add_options --host=aarch64-unknown-linux-gnu
ac_add_options --enable-application=comm/mail
ac_add_options --disable-updater
ac_add_options --disable-crashreporter
ac_add_options --disable-tests
mk_add_options AUTOCLOBBER=1
MOZ_REQUIRE_SIGNING=
MOZ_REQUIRE_ADDON_SIGNING=0
ac_add_options --enable-official-branding
export MOZ_DEBUG_SYMBOLS=1
export MOZILLA_OFFICIAL=1
export MOZ_TELEMETRY_REPORTING= # No telemetry.
export MOZ_NO_PIE_COMPAT=1
mk_add_options MOZ_MAKE_FLAGS=-j16
cd ~/ ( ~/ is /home/ubuntu )
mkdir build
mkdir alternatives
cd build
hg clone https://hg.mozilla.org/mozilla-central
(Nb: latest commit was aebf5d6)
cp mozconfig mozilla-central/
cd mozilla-central
hg clone https://hg.mozilla.org/comm-central
(Nb: latest commit was aec43dd)
mv comm-central comm
cd ~/build/mozilla-central
./mach --no-interactive bootstrap --application-choice "Firefox for Desktop"
./mach build
mach build runs for 14 minutes until 'wasm-ld' is invoked and throws this error:
wasm-ld: error: unable to find library -lgcc
14:13.85 widget/gtk/mozwayland/libmozwayland.so
14:14.08 xpcom/build
14:17.13 wasm-ld-13: error: unable to find library -lgcc
14:17.13 clang: error: linker command failed with exit code 1 (use -v to see invocation)
14:17.13 gmake[4]: *** [/home/ubuntu/build/mozilla-central/config/rules.mk:498: rlbox.wasm] Error 1
14:17.13 gmake[3]: *** [/home/ubuntu/build/mozilla-central/config/recurse.mk:72: security/rlbox/target-objects] Error 2
14:17.13 gmake[3]: *** Waiting for unfinished jobs....
I couldn't figure out how to modify the source or use wasm flags to fix this so I created my own wasm-ld binary which I could switch in and out using update-alternatives. I used shc to create compiled executable binaries from bash scripts.
Script converted to a binary:
/usr/lib/llvm-13/bin/wasm-ld --verbose -L/home/ubuntu/alternatives "$@"
When I use this custom wasm-ld with the standard gcc 12.1 libgcc.a, I get messages saying:
wasm-ld: Loading (multiple .wasm files)
wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: _muldi3.o
I modified my libgcc.a:
ar dv libgcc.a _muldi3.o
Running ./mach build again shows the same wasm-ld messages as before, except a new file type causes the error:
wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: letf2.o
I modified my libgcc.a further:
ar dv libgcc.a _muldi3.o
Running ./mach build again shows the same wasm-ld messages as before, except a new file type causes the error:
wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: trunctfdf2.o
I modified my libgcc.a further:
ar dv libgcc.a trunctfdf2.o
Note the resulting libgcc.a now had all 3 files removed.
Running ./mach build again produces:
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(algorithm.cpp.o): undefined symbol: __lttf2
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(d2fixed.cpp.o): undefined symbol: __multi3
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc.a(utimensat.o): undefined symbol: __muloti4
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc.a(utimensat.o): undefined symbol: __muloti4
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(d2fixed.cpp.o): undefined symbol: __multi3
Then: I further modified my wasm-ld to allow undefined symbols. i.e. to be equivalent to this script:
/usr/lib/llvm-13/bin/wasm-ld --verbose -L/home/ubuntu/alternatives --allow-undefined "$@"
The following 3 lines describe the final error:
ld: /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/toolkit/library/build/../../../security/rlbox/rlbox.wasm.o: in function `w2c___intscan':
/home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c:706306: undefined reference to `Z_envZ___multi3Z_vijjjj'
/usr/bin/ld: libxul.so: hidden symbol `Z_envZ___multi3Z_vijjjj' isn't defined
I then deleted line 706306 in rlbox.wasm.c
Then ./mach build succeeds.
Reporter | ||
Updated•2 years ago
|
Reporter | ||
Comment 1•2 years ago
|
||
I tried deleting all of the lines in rlbox.wasm.c that include undefined symbols as the final step before a successful build, using sed:
sed -i ':a;N;$!ba;s/ w2c_i0 = Z_envZ___lttf2Z_ijjjj(sbx, w2c_j0, w2c_j1, w2c_j2, w2c_j3);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c
sed -i ':a;N;$!ba;s/ Z_envZ___multi3Z_vijjjj(sbx, w2c_i0, w2c_j1, w2c_j2, w2c_j3, w2c_j4);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c
sed -i ':a;N;$!ba;s/ w2c_d0 = Z_envZ___trunctfdf2Z_djj(sbx, w2c_j0, w2c_j1);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c
sed -i ':a;N;$!ba;s/ Z_envZ___muloti4Z_vijjjji(sbx, w2c_i0, w2c_j1, w2c_j2, w2c_j3, w2c_j4, w2c_i5);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c
sed -i ':a;N;$!ba;s/ Z_envZ___multi3Z_vijjjj(sbx, w2c_i0, w2c_j1, w2c_j2, w2c_j3, w2c_j4);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c
This was an alternative to just deleting line 706306 in rlbox.wasm.c (my previous final step).
The build was successful and the result of 'mach package' worked well on arm64/aarch64.
Updated•2 years ago
|
Comment 2•2 years ago
|
||
Can you submit a patch?
Reporter | ||
Comment 3•2 years ago
|
||
(In reply to Magnus Melin [:mkmelin] from comment #2)
Can you submit a patch?
Hi Magnus,
I will have a look at the possibility of submitting a patch. I'm a bit swamped with work at the moment so it won't be for a few days (sorry). I don't know if licensing is a problem with adding my solution to the source code. Mozilla has an MPL 2.0 license.
Do you happen to know if licensing allows my custom versions of wasm-ld (from the llvm project wich is Apache 2.0 license) and libgcc.a (from the gcc project has a GPL license) to be included in the mozilla-beta source code, maybe as a third party item?
I believe that all of the above licenses class themselves as open source.
Kind regards,
Daniel
Comment 4•2 years ago
|
||
If the problem is in third party libraries, I think you'll likely need to get fixes submitted to the respective upstream project.
Reporter | ||
Comment 5•2 years ago
|
||
(In reply to Magnus Melin [:mkmelin] from comment #4)
If the problem is in third party libraries, I think you'll likely need to get fixes submitted to the respective upstream project.
Hi Magnus,
I think the problem is with the mozilla beta code. I'm going to try adding flags to the wasm-ld command in ...mozilla-beta/config/rules.mk (line 498).
Cheers,
Daniel
Reporter | ||
Comment 6•2 years ago
|
||
I am now attempting to build Thunderbird 102 from the latest mozilla-beta and comm-beta repo. Unfortunately I have encountered some additional problems and have not been able to achieve a successful build yet. I will provide further advice and hopefully a patch as soon as I can.
Updated•2 years ago
|
Reporter | ||
Comment 7•2 years ago
|
||
I had a lot more problems trying to build for aarch64 Thunderbird v102 from the mozilla-beta repo than when I built the previous nightly v102 from mozilla-central.
Mozilla-beta asks you to specify a wasi-sysroot directory. I suspect the extra wasi-sysroot code creates even more problems for aarch64.
Clang throws errors about half-way in to the build because it loses the ability to locate header files (*.h). Clang searches these include directories by default:
/usr/lib/llvm-14/lib/clang/14.0.0/include
/usr/local/include
/usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/include
/usr/include/aarch64-linux-gnu
/usr/include
However, at about half-way in to the build all searches for header files in these directories fail. Everything built with clang fails unless you try to correct this.
Then wasm-ld fails with the same errors described above (i.e. the same errors as when I built v102 from mozilla-central). However, there are even more problems, including:
[1]
14:03.46 wasm-ld-14: error: cannot open crt1.o: No such file or directory
14:03.46 wasm-ld-14: error: unable to find library -lwasi-emulated-process-clocks
14:03.46 wasm-ld-14: error: unable to find library -lc++
14:03.46 wasm-ld-14: error: unable to find library -lc++abi
14:03.46 wasm-ld-14: error: unable to find library -lc
14:03.46 wasm-ld-14: error: unable to find library -lgcc
[2]
wasm-ld: warning: function signature mismatch: clock
defined as () -> i32 in affixmgr.wasm
defined as () -> i64 in wasi-sysroot/lib/wasm32-wasi/libwasi-emulated-process-clocks.a(clock.o)
00b40d2: error: unexpected opcode: 0xfc 0xa
gmake[4]: *** [.../mozilla-beta/config/rules.mk:501: rlbox.wasm.c] Error 1
gmake[3]: *** [.../mozilla-beta/config/recurse.mk:72: security/rlbox/target-objects] Error 2
I was actually able to modify the build to correct for all errors up until the "unexpected opcode". By that time I was exhausted and I gave up.
However ....
I was able to build relatively easily if I disabled wasm sandboxed libraries. The steps are:
- Add to mozconfig:
ac_add_options --without-wasm-sandboxed-libraries - EDIT:
gfx/webrender_bindings/webrender_ffi.h
Comment out line 76:
// const uint64_t ROOT_CLIP_CHAIN = ~0;
Otherwise, there is an error that says that ROOT_CLIP_CHAIN has been defined twice.
Reporter | ||
Comment 8•2 years ago
|
||
Reporter | ||
Comment 9•2 years ago
|
||
Apply aarch64.patch to the mozilla-beta
directory (referred to as TOPSRCDIR in my comment of June 17 2022 Australian Eastern Time)
$TOPSRCDIR/config/rules.mk
line 498 was changed. The WASM compiler needs to have the following options added so it can allow undefined symbols in linked binaries and also so it can link to a customlibgcc.a
:-Wl,--allow-undefined -L$(TOPSRCDIR)/aarch64-libgcc
$TOPSRCDIR/gfx/webrender_bindings/webrender_ffi.h
line 76 was commented out because the./mach build
process was giving an error that said thatROOT_CLIP_CHAIN
had already been defined. So the line 76 was commented as below:// const uint64_t ROOT_CLIP_CHAIN = ~0;
Reporter | ||
Comment 10•2 years ago
|
||
I finally had success building Thunderbird 102 beta on an aarch64 host for an aarch64 target.
I used the steps below.
My host computer was an AWS EC2 c7g.4xlarge aarch64 instance using Ubuntu Jammy AMI ami-070650c005cce4203.
All changes are to mozilla-beta (changeset sha b1bd8e0a15b5). I also built with comm-beta (changeset sha 53e7ea2f35af). These were current on June 16.
Install packages that ./mach bootstrap
does not install (on an aarch64 host):
sudo apt install git mercurial nano watchman python3-pywatchman python3 python3-pip python3-setuptools python3-wheel default-jre default-jdk wget curl gcc g++ binutils build-essential cmake ninja-build nasm libc6 libc6-dev libgcc-11-dev libstdc++-11-dev libstdc++6 linux-libc-dev libstdc++6 libstdc++-11-dev libx11-dev libxext-dev libxt-dev libxcb1-dev libxcb-shm0-dev libx11-xcb-dev libbsd-dev libgtk-3-dev libdbus-glib-1-dev
- Installed Rust (including rustc and cargo) 1.60, cbindgen 0.24.3, bindgen 0.60.1, Node 16.15.1, npm 8.12.1
- Installed LLVM 14.0.5 (for Ubuntu download the script from llvm.org then
sudo ./llvm.sh 14 all
). ./mach bootstrap
also fails to create theWASI_SYSROOT
directory, which is provided below.
Download source:
export TOPSRCDIR=/home/ubuntu/build-102/mozilla-beta
(Change this to point to your mozilla-beta directory)
cd /home/ubuntu/build-102
(Change this also)
hg clone https://hg.mozilla.org/releases/mozilla-beta
cd mozilla-beta
hg clone https://hg.mozilla.org/releases/comm-beta comm
All of my changes to mozilla-beta can be applied with the following steps:
- Apply aarch64.patch to the
mozilla-beta
directory (referred to as TOPSRCDIR below) tar -xzvf aarch64-libgcc.tar.gz -C $TOPSRCDIR
tar -xzvf aarch64-wasi-sdk.tar.gz -C $TOPSRCDIR
cp 102-build.sh $TOPSRCDIR/
I copied the compiler runtime to the corresponding LLVM directory prior to building Thunderbird:
sudo cp -p $TOPSRCDIR/aarch64-wasi-sdk/lib/clang/14.0.5/lib/wasi/libclang_rt.builtins-wasm32.a /usr/lib/llvm-14/lib/clang/14.0.5/lib/wasi/
mozconfig
Nb: the lines relevant to building Thunderbird on aarch64 are marked with a comment - #):
ac_add_options --target=aarch64-unknown-linux-gnu
#
ac_add_options --host=aarch64-unknown-linux-gnu
#
ac_add_options --enable-application=comm/mail
ac_add_options --disable-updater
ac_add_options --disable-crashreporter
ac_add_options --disable-tests
ac_add_options --enable-official-branding
mk_add_options TOPSRCDIR=/home/ubuntu/build-102/mozilla-beta
# Change this to point to your directory
mk_add_options AUTOCLOBBER=1
mk_add_options MOZ_MAKE_FLAGS=-j16
MOZ_REQUIRE_SIGNING=
MOZ_REQUIRE_ADDON_SIGNING=0
export TOPSRCDIR=/home/ubuntu/build-102/mozilla-beta
# Change this to point to your directory
export WASI_SYSROOT=$TOPSRCDIR/aarch64-wasi-sdk/share/wasi-sysroot
#
export MOZ_DEBUG_SYMBOLS=1
export MOZILLA_OFFICIAL=1
export MOZ_TELEMETRY_REPORTING=
export MOZ_NO_PIE_COMPAT=1
To build successfully, use my script (102-build.sh):
cd $TOPSRCDIR
./102-build.sh
(This succeeded on the aarch64 host machine I used)
Further explanation (How the above files and changes were created):
1. Changes recorded in the aarch64.patch file:
$TOPSRCDIR/config/rules.mk
line 498 was changed. The WASM compiler needs to have the following options added so it can allow undefined symbols in linked binaries and also so it can link to a customlibgcc.a
:-Wl,--allow-undefined -L$(TOPSRCDIR)/aarch64-libgcc
$TOPSRCDIR/gfx/webrender_bindings/webrender_ffi.h
line 76 was commented out because the./mach build
process was giving an error that said thatROOT_CLIP_CHAIN
had already been defined. So the line 76 was commented as below:// const uint64_t ROOT_CLIP_CHAIN = ~0;
2. Custom libgcc.a (from aarch64-libgcc.tar.gz
)
$TOPSRCDIR/aarch64-libgcc/libgcc.a
is a custom libgcc.a created by doing this:
mkdir -p $TOPSRCDIR/aarch64-libgcc
cp -p /usr/lib/gcc/aarch64-linux-gnu/11/libgcc.a $TOPSRCDIR/aarch64-libgcc/
cd $TOPSRCDIR/aarch64-libgcc
llvm-ar dv libgcc.a _muldi3.o
llvm-ar dv libgcc.a letf2.o
llvm-ar dv libgcc.a trunctfdf2.o
Nb: The changes to rules.mk
(in 1. first dot point above) and the custom libgcc.a (in 2. above) eliminate the following errors that occur during ./mach build
:
wasm-ld: error: unable to find library -lgcc
wasm-ld: error: unknown file type: _muldi3.o
wasm-ld: error: unknown file type: letf2.o
wasm-ld: error: unknown file type: trunctfdf2.o
wasm-ld: error: undefined symbols
(multiple lines of errors each naming different symbols)
3. The WASM compiler runtime (from aarch64-wasi-sdk.tar.gz) is:
$TOPSRCDIR/aarch64-wasi-sdk/lib/clang/14.0.5/lib/wasi/libclang_rt.builtins-wasm32.a
Nb: I copied this file to /usr/lib/llvm-14/lib/clang/14.0.5/lib/wasi/
prior to doing a build.
4. WASI_SYSROOT is the folder structure containing various files (also from aarch64-wasi-sdk.tar.gz) that is found in:
$TOPSRCDIR/aarch64-wasi-sdk/share/wasi-sysroot
5. Method that I used to build the WASM COMPILER RUNTIME AND WASI_SYSROOT:
These following 3 files were a guide. Nb: These files were not created by me. They are included with the mozilla-beta source:
$TOPSRCDIR/taskcluster/scripts/misc/build-compiler-rt-wasi.sh
$TOPSRCDIR/taskcluster/scripts/misc/build-sysroot-wasi.sh
$TOPSRCDIR/taskcluster/ci/fetch/toolchains.yml
./mach build
will fail if you use the incorrect sha/changeset for wasi-sdk with the git checkout
command OR the incorrect sha/changeset for llvm-project with the 'git checkout` command:
git clone https://github.com/WebAssembly/wasi-sdk.git
cd wasi-sdk
git checkout 37ae6af88201884c8b98a65adcd6d8cd1284f95e
(The above sha is the correct changeset to use. It is fromtoolchains.yml
)git submodule update --init
(The above command downloads llvm-project source code)- The llvm-project source version downloaded above (by
git submodule update --init
) is version 14.0.3. - You cannot use llvm-project source version 14.0.3 with LLVM 14.0.5 (which is my LLVM version).
mach ./build
will ultimately fail because the WASI_SYSROOT directories and files were created from the use of an llvm-project source (14.0.3) that is incompatible with the system's installed LLVM (in my case: 14.0.5). - Also you cannot use llvm-project source version 14.0.3 with LLVM 14.0.0 (which is the default LLVM available from the Ubuntu repository).
- I needed to set the llvm-project source to the correct version (14.0.5 in my case) with the following commands:
cd src/llvm-project
git checkout c12386ae247c0d46e1d513942e322e3a0510b126
(This is the correct sha/changeset for llvm 14.0.5. If using a different version of LLVM, you will need to change the sha/changeset to mach your version of llvm).
Then follow the guides inbuild-compiler-rt-wasi.sh
andbuild-sysroot-wasi.sh
.
6. 102-build.sh script to automate deleting lines from rlbox.wasm.c
I was unable to figure out how to run a build without encountering errors because of undefined references in rlbox.wasm.c
. There were 3 undefined references, each occurring multiple times:
Z_envZ___cxa_allocate_exceptionZ_ii
Z_envZ___cxa_throwZ_viii
Z_envZ___multi3Z_vijjjj
I needed the following 3 steps for a successful build:
- I issued the command
./mach build
. This ran for 37 minutes before throwing multiple errors, all indicating one of the above 3 undefined references. All errors looked like the following, except with varying function names (in bold below) and varying symbol names (also in bold below):
/usr/bin/ld:
/home/ubuntu/build-102/mozilla-beta/obj-aarch64-unknown-linux-gnu/toolkit/library/build/../../../security/rlbox/
rlbox.wasm.o:
in function
w2c_operator_new_unsigned_long_
':
/home/ubuntu/build-102/mozilla-beta/obj-aarch64-unknown-linux-gnu/security/rlbox/
rlbox.wasm.c
:686435:
undefined reference
to
Z_envZ___cxa_allocate_exceptionZ_ii'
/usr/bin/ld: libxul.so: hidden symbol Z_envZ___cxa_throwZ_viii isn't defined
`/usr/bin/ld: final link failed: bad value
`- Then there are multiple similar lines like the above. Then ...
clang: error: linker command failed
with exit code 1 (use -v to see invocation)
gmake[4]: [/home/ubuntu/build-102/mozilla-beta/config/
rules.mk:531: libxul.so
] Error 1
gmake[3]: [/home/ubuntu/build-102/mozilla-beta/config/
recurse.mk:72: toolkit/library/build/target
] Error 2
gmake[2]: [/home/ubuntu/build-102/mozilla-beta/config/
recurse.mk:34: compile
] Error 2
gmake[1]: [/home/ubuntu/build-102/mozilla-beta/config/
rules.mk:352: default
] Error 2
gmake: [
client.mk:63: build
] Error 2
- After the above error, the build stopped. I then deleted the lines in rlbox.wasm.c containing undefined references using the sed command (see contents of '102-build.sh' for exact sed commands).
- I then ran
./mach build
again. This time the build succeeded.
- I wrote the "102-build.sh" script to automate the 3 steps above.
Reporter | ||
Comment 11•2 years ago
|
||
Use my pre-made libgcc.a
tar -xzvf aarch64-libgcc.tar.gz -C $TOPSRCDIR
OR use the method below to create your own custom libgcc.a
$TOPSRCDIR/aarch64-libgcc/libgcc.a
is a custom libgcc.a created by doing this:
mkdir -p $TOPSRCDIR/aarch64-libgcc
cp -p /usr/lib/gcc/aarch64-linux-gnu/11/libgcc.a $TOPSRCDIR/aarch64-libgcc/
cd $TOPSRCDIR/aarch64-libgcc
llvm-ar dv libgcc.a _muldi3.o
llvm-ar dv libgcc.a letf2.o
llvm-ar dv libgcc.a trunctfdf2.o
My change to $TOPSRCDIR/config/rules.mk
line 498 allows wasm-ld
to link the above custom libgcc.a. The WASM compiler needs to have the following options added so it can allow undefined symbols in linked binaries and also so it can link to a custom libgcc.a
. Prior to this, an error was thrown because wasm-ld
failed to find -lgcc
. So I added to rules.mk
line 498:
+ -Wl,--allow-undefined -L$(TOPSRCDIR)/aarch64-libgcc
Reporter | ||
Comment 12•2 years ago
|
||
Use my pre-made WASM compiler runtime and WASI_SYSROOT files and directories:
tar -xzvf aarch64-wasi-sdk.tar.gz -C $TOPSRCDIR
OR Use the method below to create your own WASM compiler runtime and WASI_SYSROOT:
-
The WASM compiler runtime (from aarch64-wasi-sdk.tar.gz) is:
$TOPSRCDIR/aarch64-wasi-sdk/lib/clang/14.0.5/lib/wasi/libclang_rt.builtins-wasm32.a
Nb: I copied this file to/usr/lib/llvm-14/lib/clang/14.0.5/lib/wasi/
prior to doing a build. -
WASI_SYSROOT is the folder structure containing various files (also from aarch64-wasi-sdk.tar.gz) that is found in:
$TOPSRCDIR/aarch64-wasi-sdk/share/wasi-sysroot
These following 3 files were a guide. Nb: These files were not created by me. They are included with the mozilla-beta source:
$TOPSRCDIR/taskcluster/scripts/misc/build-compiler-rt-wasi.sh
$TOPSRCDIR/taskcluster/scripts/misc/build-sysroot-wasi.sh
$TOPSRCDIR/taskcluster/ci/fetch/toolchains.yml
./mach build
will fail if you use the incorrect sha/changeset for wasi-sdk with the git checkout
command OR the incorrect sha/changeset for llvm-project with the 'git checkout` command:
git clone https://github.com/WebAssembly/wasi-sdk.git
cd wasi-sdk
git checkout 37ae6af88201884c8b98a65adcd6d8cd1284f95e
(The above sha is the correct changeset to use. It is fromtoolchains.yml
)git submodule update --init
(The above command downloads llvm-project source code)- The llvm-project source version downloaded above (by
git submodule update --init
) is version 14.0.3. - You cannot use llvm-project source version 14.0.3 with LLVM 14.0.5 (which is my LLVM version).
mach ./build
will ultimately fail because the WASI_SYSROOT directories and files were created from the use of an llvm-project source (14.0.3) that is incompatible with the system's installed LLVM (in my case: 14.0.5). - Also you cannot use llvm-project source version 14.0.3 with LLVM 14.0.0 (which is the default LLVM available from the Ubuntu repository).
- I needed to set the llvm-project source to the correct version (14.0.5 in my case) with the following commands:
cd src/llvm-project
git checkout c12386ae247c0d46e1d513942e322e3a0510b126
(This is the correct sha/changeset for llvm 14.0.5. If using a different version of LLVM, you will need to change the sha/changeset to mach your version of llvm).
Then follow the guides inbuild-compiler-rt-wasi.sh
andbuild-sysroot-wasi.sh
Reporter | ||
Comment 13•2 years ago
|
||
102-build.sh automates all of the following (below):
I was unable to figure out how to run a build without encountering errors because of undefined references in rlbox.wasm.c
. There were 3 undefined references, each occurring multiple times:
Z_envZ___cxa_allocate_exceptionZ_ii
Z_envZ___cxa_throwZ_viii
Z_envZ___multi3Z_vijjjj
I needed the following 3 steps for a successful build:
- I issued the command
./mach build
. This ran for 37 minutes before throwing multiple errors, all indicating one of the above 3 undefined references. All errors looked like the following, except with varying function names (in bold below) and varying symbol names (also in bold below):
/usr/bin/ld:
/home/ubuntu/build-102/mozilla-beta/obj-aarch64-unknown-linux-gnu/toolkit/library/build/../../../security/rlbox/
rlbox.wasm.o:
in function
w2c_operator_new_unsigned_long_
':
/home/ubuntu/build-102/mozilla-beta/obj-aarch64-unknown-linux-gnu/security/rlbox/
rlbox.wasm.c
:686435:
undefined reference
to
Z_envZ___cxa_allocate_exceptionZ_ii'
/usr/bin/ld: libxul.so: hidden symbol Z_envZ___cxa_throwZ_viii isn't defined
`/usr/bin/ld: final link failed: bad value
`- Then there are multiple similar lines like the above. Then ...
clang: error: linker command failed
with exit code 1 (use -v to see invocation)
gmake[4]: [/home/ubuntu/build-102/mozilla-beta/config/
rules.mk:531: libxul.so
] Error 1
gmake[3]: [/home/ubuntu/build-102/mozilla-beta/config/
recurse.mk:72: toolkit/library/build/target
] Error 2
gmake[2]: [/home/ubuntu/build-102/mozilla-beta/config/
recurse.mk:34: compile
] Error 2
gmake[1]: [/home/ubuntu/build-102/mozilla-beta/config/
rules.mk:352: default
] Error 2
gmake: [
client.mk:63: build
] Error 2
- After the above error, the build stopped. I then deleted the lines in rlbox.wasm.c containing undefined references using the sed command (see contents of '102-build.sh' for exact sed commands).
- I then ran
./mach build
again. This time the build succeeded.
- I wrote the "102-build.sh" script to automate the 3 steps above.
Reporter | ||
Comment 14•2 years ago
|
||
This is the mozconfig
file that I used.
Reporter | ||
Updated•2 years ago
|
Reporter | ||
Updated•2 years ago
|
Reporter | ||
Comment 15•2 years ago
|
||
Forgot to mention: Please change the first line in my 102-build.sh
script to suit your system's directories:
export TOPSRCDIR=/home/ubuntu/build-102/mozilla-beta
Comment 16•2 years ago
|
||
(In reply to Dan from comment #9)
$TOPSRCDIR/gfx/webrender_bindings/webrender_ffi.h
line 76 was commented out because the./mach build
process was giving an error that said thatROOT_CLIP_CHAIN
had already been defined. So the line 76 was commented as below:
// const uint64_t ROOT_CLIP_CHAIN = ~0;
We see that error as well when building from mozilla-beta/comm-beta for Windows locally. However, Thunderbird beta is built in automation
https://treeherder.mozilla.org/jobs?repo=comm-beta
and doesn't appear to have the issue, so they must be doing something differently. BTW, we commented out webrender_ffi_generated.h line #24.
Reporter | ||
Comment 17•2 years ago
|
||
Hi Rachel / BB,
I am currently building Thunderbird version 91 for aarch64. I noticed that I now need to comment out the line involving ROOT_CLIP_CHAIN
in mozilla-esr91 as below:
cd ~/mozilla-esr91
nano gfx/webrender_bindings/webrender_ffi.h
Comment out line 80:// const uint64_t ROOT_CLIP_CHAIN = ~0;
I think the last time I built from mozilla-esr91 was 2 weeks ago. I did not need to make this change back then when building.
Comment 18•2 years ago
|
||
Related to bug 1773259, apparently using a newer version of cbingen than before causes the issue.
Reporter | ||
Comment 19•2 years ago
|
||
Thanks Rachel,
I'm leaving my system with the latest cbindgen, and just applying my patch prior to each build. It's an interesting debate: whether to install the updated cbindgen on the official build systems (which would require updating the source in gfx/webrender_bindings/webrender_ffi.h).
All the best,
Daniel
Reporter | ||
Updated•2 years ago
|
Reporter | ||
Updated•2 years ago
|
Reporter | ||
Comment 20•2 years ago
|
||
Hi,
The method (described above) reliably builds a Thunderbird 102 release version (aarch64 Linux) from the mozilla-esr102 and comm-esr102 repositories.
All the best,
Daniel
Updated•2 years ago
|
Reporter | ||
Updated•2 years ago
|
Reporter | ||
Comment 22•2 years ago
|
||
Hi,
The cbindgen (ROOT_CLIP_CHAIN) issue was just a minor item mentioned in this bug report. This bug report is not a duplicate of the cbindgen (ROOT_CLIP_CHAIN) issue. The main issue in my bug report was the WASM related errors that occurred during building Thunderbird 102 for Linux aarch64 on a Linux aarch64 host (i.e. I do not cross-compile). These WASM related erros still occcur when building Thunderbird 102 for Linux aarch64 on a Linux aarch64 host right now. I checked this when I did a build just a few hours ago.
I have also noticed that almost identical WASM related errors occured when I tried building Thunderbird 102 for Windows using MozillaBuild on a Windows 10 host (Nb: The host was a VMware Workstation virtual machine running Windows 10 Pro). i.e. I did not cross-compille. Therefore this bug report is probably relevant to architectures other than Linux aarch64.
The method described above (deleting undefined references from rlbox.wasm.c) allows both the Linux aarch64 and Windows builds described above to complete successfully. I have not tried building with other architectures. The method described above is messy because the build fails, then I delete the offending lines in rlbox.wasm.c, and then I recommence the build (which succeeds the second time around). There should be a way to fix this so the build just succeeds the first time. I am still trying different methods to come up with a solution.
I recently tried:
- For aarch64 builds from an aarch64 host: I tried using a wasi-sysroot created with LLVM-14.0.3 and build using LLVM-14.03. This did not resolve the error. I plan to try this in the next few days: wasi-sysroot created with LLVM-14.0.4 and build using LLVM-14.04.
- For Windows builds from a Windows host: I plan to try this in the next few days: wasi-sysroot created with LLVM-14.04. Nb: ./mach bootstrap uses LLVM-14.04 with MozillaBiuld on a Windows host.
All the best,
Daniel
Comment 23•2 years ago
|
||
Sorry about the mis-filing.
Our CI does run a Windows build on Windows hosts targeting x64 at least once daily - but only on comm-central. I just started up one-off on comm-esr102; we shall see how it fares.
Maybe you mentioned this above, but is there a reason why you are not cross compiling? Aarch64 systems are not a supported build host (with the exception of Mac M1s). https://firefox-source-docs.mozilla.org/build/buildsystem/supported-configurations.html#supported-build-hosts
Linux-aarch64 is a Tier 2 cross-compile supported target. Builds are run at least once daily on comm-central (and I personally run that build on a Raspi400).
Reporter | ||
Comment 24•2 years ago
|
||
Hi Rob,
No worries. Thanks for the info, particularly on supported build hosts. I must admit that my first trial of building aarch64 Linux targets was cross-compiling using an Ubuntu host. However, I ran into some problems and ended up successfully building on an aarch64 host. Thunderbird esr91 did not have the rlbox wasm feature and therefore I did not run into any of the problems in this bug report.
All the best,
Daniel
Comment 25•2 years ago
|
||
Mind if I close this then?
Reporter | ||
Comment 26•2 years ago
|
||
Hi Rob,
My builds are working well, so I don't mind if you close this report. I'll open a bug if I find that I can't build anymore.
All the best,
Daniel
Just for future reference:
I tried building the latest from mozilla-central and comm-central. I still get the same difficulties.
I tried changing LLVM minor versions (14.0.3, 14.0.4 and 14.0.6). I built the wasi compiler-rt and wasi-sysroot on each occasion with each corresponding respective LLVM minor version. All of this did not make a difference. The wasm related errors still occurred.
I noticed that:
obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.h
contains:
extern u32 Z_envZ___cxa_allocate_exceptionZ_ii(void*, u32);
extern void Z_envZ___cxa_throwZ_viii(void*, u32, u32, u32);
extern void Z_envZ___multi3Z_vijjjj(void*, u32, u64, u64, u64, u64);
However, these definitions are not carried through to rlbox.wasm.c despite the include statement.
I tried adding these definitions to rlbox.wasm.c
However, the build did not proceed successfully.
Updated•2 years ago
|
Description
•