Closed Bug 1771649 Opened 2 years ago Closed 2 years ago

Building Thunderbird 102 on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments June 16 PDT)

Categories

(Thunderbird :: Build Config, defect)

Thunderbird 102
ARM64
Linux
defect

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: daniel.bonner, Unassigned)

References

(Blocks 1 open bug)

Details

Attachments

(6 files)

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:100.0) Gecko/20100101 Firefox/100.0

Steps to reproduce:

Building 102.0a1 on aarch64 arm64 has wasm-ld errors

Summary:

I am running Amazon EC2 instance (ami-078d455b3b6ad0d4e, Ubuntu Jammy, aarch64) with the aim of building Thunderbird 91 and Thunderbird 102, both outputting aarch64 packages.

Note: Thunderbird 91 builds OK.

Note: Thunderbird 102 has the following wasm-ld errors during the "./mach build" process. When one of these errors is fixed, I could re-run the build process until the next wasm-ld error occurred:

  1. wasm-ld: error: unable to find library -lgcc

  2. wasm-ld: error: unknown file type: _muldi3.o

  3. wasm-ld: error: unknown file type: letf2.o

  4. wasm-ld: error: unknown file type: trunctfdf2.o

  5. wasm-ld: error: undefined symbols (8 lines of errors each naming different symbols)

  6. Finally ld throws an error from rlbox.wasm.c

I am able to fix the wasm-ld errors by:

A. Forcing wasm-ld to have these default flags:

-L(directory containing a custom libgcc.a)
--allow-undefined

B. Removing the unknown file types from my custom libgcc.a :

ar dv libgcc.a _muldi3.o
ar dv libgcc.a letf2.o
ar dv libgcc.a trunctfdf2.o

C. Deleting line 706306 in rlbox.wasm.c (although technically this is an ld not wasm-ld error)

After implementing fixes (A to C), the Thunderbird 102 build succeeds. The result from ./mach package works perfectly (tested on aarch64 devices).

I set up the system for new builds of Thunderbird 102 by keeping my custom wasm-ld binary selected with update-alternatives and making it use a custom libgcc.a

However, I need to wait until the Thunderbird 102 build produces the final error (rlbox.wasm.c:706306: undefined reference to `Z_envZ___multi3Z_vijjjj') before I can delete the offending line 706306. Finally, I rerun ./mach build to a successful build.

Detailed steps:

Note: I use llvm-13 and gcc 12.1 below. However, I have also tried llvm-10, llvm-13, llvm-14 and llvm-15 (nightly) combined with gcc-9.3, gcc-11.2 and gcc-12.1. The same errors occur in multiple combinations of gcc and llvm.

sudo apt update && sudo apt upgrade

sudo apt install mlocate git mercurial python3 python3-pip ansible nano watchman python3 python3-pip python3-setuptools python3-wheel default-jre default-jdk gcc g++ binutils libc6 libc6-dev libgcc-11-dev libstdc++-11-dev libstdc++6 linux-libc-dev libstdc++6 libstdc++-11-dev libx11-dev libxext-dev libxt-dev libxcb1-dev libxcb-shm0-dev libx11-xcb-dev clang-13 clang-tools-13 clang-format-13 clangd-13 clang-tidy-13 libclang-13-dev libclang-common-13-dev libclang-cpp13 libclang1-13 libclang-cpp13-dev llvm-13 llvm-13-runtime libllvm13 llvm-13-dev libc++1 libc++abi1 libc++-dev libgtk-3-dev libdbus-glib-1-dev wget curl git gcc g++ python3 python3-pip build-essential cmake ninja-build fonts-powerline powerline build-essential python3-pip cmake ninja-build python3 libc++1-13 libc++abi1-13 libunwind-13 lld-13 libbsd-dev emscripten wasi-libc libunwind-13 libunwind-13-dev libc++abi-13-dev libc++-13-dev gcc-12 g++-12 shc

Install Rust 1.61 and 1.53 with:
./rustup-init.sh -y --default-toolchain stable --default-host aarch64-unknown-linux-gnu --component rustfmt

Install cbindgen 0.23.0 with:
cargo install cbindgen

Install NodeJS 16.15.0 with nvm:
nvm install 16.15.0
nvm use 16.15.0
nvm alias default 16.15.0

Install npm 8.11.0 with:
npm install -g npm@latest

Contents of mozconfig:

ac_add_options --target=aarch64-unknown-linux-gnu
ac_add_options --host=aarch64-unknown-linux-gnu
ac_add_options --enable-application=comm/mail
ac_add_options --disable-updater
ac_add_options --disable-crashreporter
ac_add_options --disable-tests
mk_add_options AUTOCLOBBER=1
MOZ_REQUIRE_SIGNING=
MOZ_REQUIRE_ADDON_SIGNING=0
ac_add_options --enable-official-branding
export MOZ_DEBUG_SYMBOLS=1
export MOZILLA_OFFICIAL=1
export MOZ_TELEMETRY_REPORTING= # No telemetry.
export MOZ_NO_PIE_COMPAT=1
mk_add_options MOZ_MAKE_FLAGS=-j16

cd ~/ ( ~/ is /home/ubuntu )
mkdir build
mkdir alternatives
cd build
hg clone https://hg.mozilla.org/mozilla-central
(Nb: latest commit was aebf5d6)
cp mozconfig mozilla-central/
cd mozilla-central
hg clone https://hg.mozilla.org/comm-central
(Nb: latest commit was aec43dd)
mv comm-central comm
cd ~/build/mozilla-central
./mach --no-interactive bootstrap --application-choice "Firefox for Desktop"
./mach build

mach build runs for 14 minutes until 'wasm-ld' is invoked and throws this error:
wasm-ld: error: unable to find library -lgcc

14:13.85 widget/gtk/mozwayland/libmozwayland.so
14:14.08 xpcom/build
14:17.13 wasm-ld-13: error: unable to find library -lgcc
14:17.13 clang: error: linker command failed with exit code 1 (use -v to see invocation)
14:17.13 gmake[4]: *** [/home/ubuntu/build/mozilla-central/config/rules.mk:498: rlbox.wasm] Error 1
14:17.13 gmake[3]: *** [/home/ubuntu/build/mozilla-central/config/recurse.mk:72: security/rlbox/target-objects] Error 2
14:17.13 gmake[3]: *** Waiting for unfinished jobs....

I couldn't figure out how to modify the source or use wasm flags to fix this so I created my own wasm-ld binary which I could switch in and out using update-alternatives. I used shc to create compiled executable binaries from bash scripts.

Script converted to a binary:
/usr/lib/llvm-13/bin/wasm-ld --verbose -L/home/ubuntu/alternatives "$@"

When I use this custom wasm-ld with the standard gcc 12.1 libgcc.a, I get messages saying:
wasm-ld: Loading (multiple .wasm files)
wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: _muldi3.o

I modified my libgcc.a:
ar dv libgcc.a _muldi3.o

Running ./mach build again shows the same wasm-ld messages as before, except a new file type causes the error:

wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: letf2.o

I modified my libgcc.a further:
ar dv libgcc.a _muldi3.o

Running ./mach build again shows the same wasm-ld messages as before, except a new file type causes the error:

wasm-ld: Processing (multiple .wasm files)
wasm-ld: Processing: /home/ubuntu/alternatives/libgcc.a
wasm-ld: error: unknown file type: trunctfdf2.o

I modified my libgcc.a further:
ar dv libgcc.a trunctfdf2.o
Note the resulting libgcc.a now had all 3 files removed.

Running ./mach build again produces:
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(algorithm.cpp.o): undefined symbol: __lttf2
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(d2fixed.cpp.o): undefined symbol: __multi3
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(charconv.cpp.o): undefined symbol: __trunctfdf2
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc.a(utimensat.o): undefined symbol: __muloti4
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc.a(utimensat.o): undefined symbol: __muloti4
0:02.89 wasm-ld: error: /home/ubuntu/.mozbuild/sysroot-wasm32-wasi/lib/wasm32-wasi/libc++.a(d2fixed.cpp.o): undefined symbol: __multi3

Then: I further modified my wasm-ld to allow undefined symbols. i.e. to be equivalent to this script:
/usr/lib/llvm-13/bin/wasm-ld --verbose -L/home/ubuntu/alternatives --allow-undefined "$@"

The following 3 lines describe the final error:

ld: /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/toolkit/library/build/../../../security/rlbox/rlbox.wasm.o: in function `w2c___intscan':

/home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c:706306: undefined reference to `Z_envZ___multi3Z_vijjjj'

/usr/bin/ld: libxul.so: hidden symbol `Z_envZ___multi3Z_vijjjj' isn't defined

I then deleted line 706306 in rlbox.wasm.c

Then ./mach build succeeds.

Summary: Building 102.0a1 on aarch64 arm64 has wasm-ld errors → Building Thunderbird 102.0a1 on aarch64 arm64 is interrupted by wasm-ld errors

I tried deleting all of the lines in rlbox.wasm.c that include undefined symbols as the final step before a successful build, using sed:

sed -i ':a;N;$!ba;s/ w2c_i0 = Z_envZ___lttf2Z_ijjjj(sbx, w2c_j0, w2c_j1, w2c_j2, w2c_j3);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c

sed -i ':a;N;$!ba;s/ Z_envZ___multi3Z_vijjjj(sbx, w2c_i0, w2c_j1, w2c_j2, w2c_j3, w2c_j4);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c

sed -i ':a;N;$!ba;s/ w2c_d0 = Z_envZ___trunctfdf2Z_djj(sbx, w2c_j0, w2c_j1);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c

sed -i ':a;N;$!ba;s/ Z_envZ___muloti4Z_vijjjji(sbx, w2c_i0, w2c_j1, w2c_j2, w2c_j3, w2c_j4, w2c_i5);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c

sed -i ':a;N;$!ba;s/ Z_envZ___multi3Z_vijjjj(sbx, w2c_i0, w2c_j1, w2c_j2, w2c_j3, w2c_j4);\n//g' /home/ubuntu/build/mozilla-central/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c

This was an alternative to just deleting line 706306 in rlbox.wasm.c (my previous final step).

The build was successful and the result of 'mach package' worked well on arm64/aarch64.

Blocks: tb102found

Can you submit a patch?

(In reply to Magnus Melin [:mkmelin] from comment #2)

Can you submit a patch?

Hi Magnus,
I will have a look at the possibility of submitting a patch. I'm a bit swamped with work at the moment so it won't be for a few days (sorry). I don't know if licensing is a problem with adding my solution to the source code. Mozilla has an MPL 2.0 license.
Do you happen to know if licensing allows my custom versions of wasm-ld (from the llvm project wich is Apache 2.0 license) and libgcc.a (from the gcc project has a GPL license) to be included in the mozilla-beta source code, maybe as a third party item?
I believe that all of the above licenses class themselves as open source.
Kind regards,
Daniel

If the problem is in third party libraries, I think you'll likely need to get fixes submitted to the respective upstream project.

(In reply to Magnus Melin [:mkmelin] from comment #4)

If the problem is in third party libraries, I think you'll likely need to get fixes submitted to the respective upstream project.

Hi Magnus,
I think the problem is with the mozilla beta code. I'm going to try adding flags to the wasm-ld command in ...mozilla-beta/config/rules.mk (line 498).
Cheers,
Daniel

I am now attempting to build Thunderbird 102 from the latest mozilla-beta and comm-beta repo. Unfortunately I have encountered some additional problems and have not been able to achieve a successful build yet. I will provide further advice and hopefully a patch as soon as I can.

Component: Untriaged → Build Config

I had a lot more problems trying to build for aarch64 Thunderbird v102 from the mozilla-beta repo than when I built the previous nightly v102 from mozilla-central.
Mozilla-beta asks you to specify a wasi-sysroot directory. I suspect the extra wasi-sysroot code creates even more problems for aarch64.
Clang throws errors about half-way in to the build because it loses the ability to locate header files (*.h). Clang searches these include directories by default:
/usr/lib/llvm-14/lib/clang/14.0.0/include
/usr/local/include
/usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../aarch64-linux-gnu/include
/usr/include/aarch64-linux-gnu
/usr/include
However, at about half-way in to the build all searches for header files in these directories fail. Everything built with clang fails unless you try to correct this.
Then wasm-ld fails with the same errors described above (i.e. the same errors as when I built v102 from mozilla-central). However, there are even more problems, including:
[1]
14:03.46 wasm-ld-14: error: cannot open crt1.o: No such file or directory
14:03.46 wasm-ld-14: error: unable to find library -lwasi-emulated-process-clocks
14:03.46 wasm-ld-14: error: unable to find library -lc++
14:03.46 wasm-ld-14: error: unable to find library -lc++abi
14:03.46 wasm-ld-14: error: unable to find library -lc
14:03.46 wasm-ld-14: error: unable to find library -lgcc
[2]
wasm-ld: warning: function signature mismatch: clock

defined as () -> i32 in affixmgr.wasm
defined as () -> i64 in wasi-sysroot/lib/wasm32-wasi/libwasi-emulated-process-clocks.a(clock.o)
00b40d2: error: unexpected opcode: 0xfc 0xa
gmake[4]: *** [.../mozilla-beta/config/rules.mk:501: rlbox.wasm.c] Error 1
gmake[3]: *** [.../mozilla-beta/config/recurse.mk:72: security/rlbox/target-objects] Error 2
I was actually able to modify the build to correct for all errors up until the "unexpected opcode". By that time I was exhausted and I gave up.
However ....
I was able to build relatively easily if I disabled wasm sandboxed libraries. The steps are:

  1. Add to mozconfig:
    ac_add_options --without-wasm-sandboxed-libraries
  2. EDIT:
    gfx/webrender_bindings/webrender_ffi.h
    Comment out line 76:
    // const uint64_t ROOT_CLIP_CHAIN = ~0;
    Otherwise, there is an error that says that ROOT_CLIP_CHAIN has been defined twice.
Attached patch aarch64.patchSplinter Review

Apply aarch64.patch to the mozilla-beta directory (referred to as TOPSRCDIR in my comment of June 17 2022 Australian Eastern Time)

  • $TOPSRCDIR/config/rules.mk line 498 was changed. The WASM compiler needs to have the following options added so it can allow undefined symbols in linked binaries and also so it can link to a custom libgcc.a:
    • -Wl,--allow-undefined -L$(TOPSRCDIR)/aarch64-libgcc
  • $TOPSRCDIR/gfx/webrender_bindings/webrender_ffi.h line 76 was commented out because the ./mach build process was giving an error that said that ROOT_CLIP_CHAIN had already been defined. So the line 76 was commented as below:
    • // const uint64_t ROOT_CLIP_CHAIN = ~0;

I finally had success building Thunderbird 102 beta on an aarch64 host for an aarch64 target.
I used the steps below.
My host computer was an AWS EC2 c7g.4xlarge aarch64 instance using Ubuntu Jammy AMI ami-070650c005cce4203.
All changes are to mozilla-beta (changeset sha b1bd8e0a15b5). I also built with comm-beta (changeset sha 53e7ea2f35af). These were current on June 16.

Install packages that ./mach bootstrap does not install (on an aarch64 host):

  1. sudo apt install git mercurial nano watchman python3-pywatchman python3 python3-pip python3-setuptools python3-wheel default-jre default-jdk wget curl gcc g++ binutils build-essential cmake ninja-build nasm libc6 libc6-dev libgcc-11-dev libstdc++-11-dev libstdc++6 linux-libc-dev libstdc++6 libstdc++-11-dev libx11-dev libxext-dev libxt-dev libxcb1-dev libxcb-shm0-dev libx11-xcb-dev libbsd-dev libgtk-3-dev libdbus-glib-1-dev
  2. Installed Rust (including rustc and cargo) 1.60, cbindgen 0.24.3, bindgen 0.60.1, Node 16.15.1, npm 8.12.1
  3. Installed LLVM 14.0.5 (for Ubuntu download the script from llvm.org then sudo ./llvm.sh 14 all).
  4. ./mach bootstrap also fails to create the WASI_SYSROOT directory, which is provided below.

Download source:

export TOPSRCDIR=/home/ubuntu/build-102/mozilla-beta (Change this to point to your mozilla-beta directory)
cd /home/ubuntu/build-102 (Change this also)
hg clone https://hg.mozilla.org/releases/mozilla-beta
cd mozilla-beta
hg clone https://hg.mozilla.org/releases/comm-beta comm

All of my changes to mozilla-beta can be applied with the following steps:

  • Apply aarch64.patch to the mozilla-beta directory (referred to as TOPSRCDIR below)
  • tar -xzvf aarch64-libgcc.tar.gz -C $TOPSRCDIR
  • tar -xzvf aarch64-wasi-sdk.tar.gz -C $TOPSRCDIR
  • cp 102-build.sh $TOPSRCDIR/

I copied the compiler runtime to the corresponding LLVM directory prior to building Thunderbird:

sudo cp -p $TOPSRCDIR/aarch64-wasi-sdk/lib/clang/14.0.5/lib/wasi/libclang_rt.builtins-wasm32.a /usr/lib/llvm-14/lib/clang/14.0.5/lib/wasi/

mozconfig

Nb: the lines relevant to building Thunderbird on aarch64 are marked with a comment - #):
ac_add_options --target=aarch64-unknown-linux-gnu #
ac_add_options --host=aarch64-unknown-linux-gnu #
ac_add_options --enable-application=comm/mail
ac_add_options --disable-updater
ac_add_options --disable-crashreporter
ac_add_options --disable-tests
ac_add_options --enable-official-branding
mk_add_options TOPSRCDIR=/home/ubuntu/build-102/mozilla-beta # Change this to point to your directory
mk_add_options AUTOCLOBBER=1
mk_add_options MOZ_MAKE_FLAGS=-j16
MOZ_REQUIRE_SIGNING=
MOZ_REQUIRE_ADDON_SIGNING=0
export TOPSRCDIR=/home/ubuntu/build-102/mozilla-beta # Change this to point to your directory
export WASI_SYSROOT=$TOPSRCDIR/aarch64-wasi-sdk/share/wasi-sysroot #
export MOZ_DEBUG_SYMBOLS=1
export MOZILLA_OFFICIAL=1
export MOZ_TELEMETRY_REPORTING=
export MOZ_NO_PIE_COMPAT=1

To build successfully, use my script (102-build.sh):

cd $TOPSRCDIR
./102-build.sh (This succeeded on the aarch64 host machine I used)

Further explanation (How the above files and changes were created):

1. Changes recorded in the aarch64.patch file:

  • $TOPSRCDIR/config/rules.mk line 498 was changed. The WASM compiler needs to have the following options added so it can allow undefined symbols in linked binaries and also so it can link to a custom libgcc.a:
    • -Wl,--allow-undefined -L$(TOPSRCDIR)/aarch64-libgcc
  • $TOPSRCDIR/gfx/webrender_bindings/webrender_ffi.h line 76 was commented out because the ./mach build process was giving an error that said that ROOT_CLIP_CHAIN had already been defined. So the line 76 was commented as below:
    • // const uint64_t ROOT_CLIP_CHAIN = ~0;

2. Custom libgcc.a (from aarch64-libgcc.tar.gz)

$TOPSRCDIR/aarch64-libgcc/libgcc.a is a custom libgcc.a created by doing this:

  • mkdir -p $TOPSRCDIR/aarch64-libgcc
  • cp -p /usr/lib/gcc/aarch64-linux-gnu/11/libgcc.a $TOPSRCDIR/aarch64-libgcc/
  • cd $TOPSRCDIR/aarch64-libgcc
  • llvm-ar dv libgcc.a _muldi3.o
  • llvm-ar dv libgcc.a letf2.o
  • llvm-ar dv libgcc.a trunctfdf2.o

Nb: The changes to rules.mk (in 1. first dot point above) and the custom libgcc.a (in 2. above) eliminate the following errors that occur during ./mach build:

  • wasm-ld: error: unable to find library -lgcc
  • wasm-ld: error: unknown file type: _muldi3.o
  • wasm-ld: error: unknown file type: letf2.o
  • wasm-ld: error: unknown file type: trunctfdf2.o
  • wasm-ld: error: undefined symbols (multiple lines of errors each naming different symbols)

3. The WASM compiler runtime (from aarch64-wasi-sdk.tar.gz) is:

$TOPSRCDIR/aarch64-wasi-sdk/lib/clang/14.0.5/lib/wasi/libclang_rt.builtins-wasm32.a
Nb: I copied this file to /usr/lib/llvm-14/lib/clang/14.0.5/lib/wasi/ prior to doing a build.

4. WASI_SYSROOT is the folder structure containing various files (also from aarch64-wasi-sdk.tar.gz) that is found in:

$TOPSRCDIR/aarch64-wasi-sdk/share/wasi-sysroot

5. Method that I used to build the WASM COMPILER RUNTIME AND WASI_SYSROOT:

These following 3 files were a guide. Nb: These files were not created by me. They are included with the mozilla-beta source:

  1. $TOPSRCDIR/taskcluster/scripts/misc/build-compiler-rt-wasi.sh
  2. $TOPSRCDIR/taskcluster/scripts/misc/build-sysroot-wasi.sh
  3. $TOPSRCDIR/taskcluster/ci/fetch/toolchains.yml

./mach build will fail if you use the incorrect sha/changeset for wasi-sdk with the git checkoutcommand OR the incorrect sha/changeset for llvm-project with the 'git checkout` command:

  • git clone https://github.com/WebAssembly/wasi-sdk.git
  • cd wasi-sdk
  • git checkout 37ae6af88201884c8b98a65adcd6d8cd1284f95e
    (The above sha is the correct changeset to use. It is from toolchains.yml)
  • git submodule update --init
    (The above command downloads llvm-project source code)
  • The llvm-project source version downloaded above (by git submodule update --init) is version 14.0.3.
  • You cannot use llvm-project source version 14.0.3 with LLVM 14.0.5 (which is my LLVM version). mach ./build will ultimately fail because the WASI_SYSROOT directories and files were created from the use of an llvm-project source (14.0.3) that is incompatible with the system's installed LLVM (in my case: 14.0.5).
  • Also you cannot use llvm-project source version 14.0.3 with LLVM 14.0.0 (which is the default LLVM available from the Ubuntu repository).
  • I needed to set the llvm-project source to the correct version (14.0.5 in my case) with the following commands:
  • cd src/llvm-project
  • git checkout c12386ae247c0d46e1d513942e322e3a0510b126 (This is the correct sha/changeset for llvm 14.0.5. If using a different version of LLVM, you will need to change the sha/changeset to mach your version of llvm).
    Then follow the guides in build-compiler-rt-wasi.sh and build-sysroot-wasi.sh.

6. 102-build.sh script to automate deleting lines from rlbox.wasm.c

I was unable to figure out how to run a build without encountering errors because of undefined references in rlbox.wasm.c. There were 3 undefined references, each occurring multiple times:

  • Z_envZ___cxa_allocate_exceptionZ_ii
  • Z_envZ___cxa_throwZ_viii
  • Z_envZ___multi3Z_vijjjj

I needed the following 3 steps for a successful build:

  1. I issued the command ./mach build. This ran for 37 minutes before throwing multiple errors, all indicating one of the above 3 undefined references. All errors looked like the following, except with varying function names (in bold below) and varying symbol names (also in bold below):
  • /usr/bin/ld:/home/ubuntu/build-102/mozilla-beta/obj-aarch64-unknown-linux-gnu/toolkit/library/build/../../../security/rlbox/rlbox.wasm.o:in functionw2c_operator_new_unsigned_long_':
    /home/ubuntu/build-102/mozilla-beta/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c:686435:undefined referencetoZ_envZ___cxa_allocate_exceptionZ_ii'
  • /usr/bin/ld: libxul.so: hidden symbol Z_envZ___cxa_throwZ_viii isn't defined`
  • /usr/bin/ld: final link failed: bad value`
  • Then there are multiple similar lines like the above. Then ...
  • clang: error: linker command failedwith exit code 1 (use -v to see invocation)
  • gmake[4]: [/home/ubuntu/build-102/mozilla-beta/config/rules.mk:531: libxul.so] Error 1
  • gmake[3]: [/home/ubuntu/build-102/mozilla-beta/config/recurse.mk:72: toolkit/library/build/target] Error 2
  • gmake[2]: [/home/ubuntu/build-102/mozilla-beta/config/recurse.mk:34: compile] Error 2
  • gmake[1]: [/home/ubuntu/build-102/mozilla-beta/config/rules.mk:352: default] Error 2
  • gmake: [client.mk:63: build] Error 2
  1. After the above error, the build stopped. I then deleted the lines in rlbox.wasm.c containing undefined references using the sed command (see contents of '102-build.sh' for exact sed commands).
  2. I then ran ./mach build again. This time the build succeeded.
  • I wrote the "102-build.sh" script to automate the 3 steps above.
Attached file aarch64-libgcc.tar.gz

Use my pre-made libgcc.a

  • tar -xzvf aarch64-libgcc.tar.gz -C $TOPSRCDIR

OR use the method below to create your own custom libgcc.a

$TOPSRCDIR/aarch64-libgcc/libgcc.a is a custom libgcc.a created by doing this:

  • mkdir -p $TOPSRCDIR/aarch64-libgcc
  • cp -p /usr/lib/gcc/aarch64-linux-gnu/11/libgcc.a $TOPSRCDIR/aarch64-libgcc/
  • cd $TOPSRCDIR/aarch64-libgcc
  • llvm-ar dv libgcc.a _muldi3.o
  • llvm-ar dv libgcc.a letf2.o
  • llvm-ar dv libgcc.a trunctfdf2.o

My change to $TOPSRCDIR/config/rules.mk line 498 allows wasm-ld to link the above custom libgcc.a. The WASM compiler needs to have the following options added so it can allow undefined symbols in linked binaries and also so it can link to a custom libgcc.a. Prior to this, an error was thrown because wasm-ld failed to find -lgcc. So I added to rules.mk line 498:
+ -Wl,--allow-undefined -L$(TOPSRCDIR)/aarch64-libgcc

Use my pre-made WASM compiler runtime and WASI_SYSROOT files and directories:

  • tar -xzvf aarch64-wasi-sdk.tar.gz -C $TOPSRCDIR

OR Use the method below to create your own WASM compiler runtime and WASI_SYSROOT:

  • The WASM compiler runtime (from aarch64-wasi-sdk.tar.gz) is:
    $TOPSRCDIR/aarch64-wasi-sdk/lib/clang/14.0.5/lib/wasi/libclang_rt.builtins-wasm32.a
    Nb: I copied this file to /usr/lib/llvm-14/lib/clang/14.0.5/lib/wasi/ prior to doing a build.

  • WASI_SYSROOT is the folder structure containing various files (also from aarch64-wasi-sdk.tar.gz) that is found in:
    $TOPSRCDIR/aarch64-wasi-sdk/share/wasi-sysroot

These following 3 files were a guide. Nb: These files were not created by me. They are included with the mozilla-beta source:

  1. $TOPSRCDIR/taskcluster/scripts/misc/build-compiler-rt-wasi.sh
  2. $TOPSRCDIR/taskcluster/scripts/misc/build-sysroot-wasi.sh
  3. $TOPSRCDIR/taskcluster/ci/fetch/toolchains.yml

./mach build will fail if you use the incorrect sha/changeset for wasi-sdk with the git checkoutcommand OR the incorrect sha/changeset for llvm-project with the 'git checkout` command:

  • git clone https://github.com/WebAssembly/wasi-sdk.git
  • cd wasi-sdk
  • git checkout 37ae6af88201884c8b98a65adcd6d8cd1284f95e
    (The above sha is the correct changeset to use. It is from toolchains.yml)
  • git submodule update --init
    (The above command downloads llvm-project source code)
  • The llvm-project source version downloaded above (by git submodule update --init) is version 14.0.3.
  • You cannot use llvm-project source version 14.0.3 with LLVM 14.0.5 (which is my LLVM version). mach ./build will ultimately fail because the WASI_SYSROOT directories and files were created from the use of an llvm-project source (14.0.3) that is incompatible with the system's installed LLVM (in my case: 14.0.5).
  • Also you cannot use llvm-project source version 14.0.3 with LLVM 14.0.0 (which is the default LLVM available from the Ubuntu repository).
  • I needed to set the llvm-project source to the correct version (14.0.5 in my case) with the following commands:
  • cd src/llvm-project
  • git checkout c12386ae247c0d46e1d513942e322e3a0510b126 (This is the correct sha/changeset for llvm 14.0.5. If using a different version of LLVM, you will need to change the sha/changeset to mach your version of llvm).
    Then follow the guides in build-compiler-rt-wasi.sh and build-sysroot-wasi.sh
Attached file 102-build.sh

102-build.sh automates all of the following (below):

I was unable to figure out how to run a build without encountering errors because of undefined references in rlbox.wasm.c. There were 3 undefined references, each occurring multiple times:

  • Z_envZ___cxa_allocate_exceptionZ_ii
  • Z_envZ___cxa_throwZ_viii
  • Z_envZ___multi3Z_vijjjj

I needed the following 3 steps for a successful build:

  1. I issued the command ./mach build. This ran for 37 minutes before throwing multiple errors, all indicating one of the above 3 undefined references. All errors looked like the following, except with varying function names (in bold below) and varying symbol names (also in bold below):
  • /usr/bin/ld:/home/ubuntu/build-102/mozilla-beta/obj-aarch64-unknown-linux-gnu/toolkit/library/build/../../../security/rlbox/rlbox.wasm.o:in functionw2c_operator_new_unsigned_long_':
    /home/ubuntu/build-102/mozilla-beta/obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.c:686435:undefined referencetoZ_envZ___cxa_allocate_exceptionZ_ii'
  • /usr/bin/ld: libxul.so: hidden symbol Z_envZ___cxa_throwZ_viii isn't defined`
  • /usr/bin/ld: final link failed: bad value`
  • Then there are multiple similar lines like the above. Then ...
  • clang: error: linker command failedwith exit code 1 (use -v to see invocation)
  • gmake[4]: [/home/ubuntu/build-102/mozilla-beta/config/rules.mk:531: libxul.so] Error 1
  • gmake[3]: [/home/ubuntu/build-102/mozilla-beta/config/recurse.mk:72: toolkit/library/build/target] Error 2
  • gmake[2]: [/home/ubuntu/build-102/mozilla-beta/config/recurse.mk:34: compile] Error 2
  • gmake[1]: [/home/ubuntu/build-102/mozilla-beta/config/rules.mk:352: default] Error 2
  • gmake: [client.mk:63: build] Error 2
  1. After the above error, the build stopped. I then deleted the lines in rlbox.wasm.c containing undefined references using the sed command (see contents of '102-build.sh' for exact sed commands).
  2. I then ran ./mach build again. This time the build succeeded.
  • I wrote the "102-build.sh" script to automate the 3 steps above.
Attached file mozconfig

This is the mozconfig file that I used.

Summary: Building Thunderbird 102.0a1 on aarch64 arm64 is interrupted by wasm-ld errors → Building Thunderbird 102 beta on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments June 17 Aus time)
Summary: Building Thunderbird 102 beta on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments June 17 Aus time) → Building Thunderbird 102 beta on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments June 17 Aus time, June 16 PDT)

Forgot to mention: Please change the first line in my 102-build.sh script to suit your system's directories:

  • export TOPSRCDIR=/home/ubuntu/build-102/mozilla-beta

(In reply to Dan from comment #9)

  • $TOPSRCDIR/gfx/webrender_bindings/webrender_ffi.h line 76 was commented out because the ./mach build process was giving an error that said that ROOT_CLIP_CHAIN had already been defined. So the line 76 was commented as below:
    • // const uint64_t ROOT_CLIP_CHAIN = ~0;

We see that error as well when building from mozilla-beta/comm-beta for Windows locally. However, Thunderbird beta is built in automation
https://treeherder.mozilla.org/jobs?repo=comm-beta
and doesn't appear to have the issue, so they must be doing something differently. BTW, we commented out webrender_ffi_generated.h line #24.

Hi Rachel / BB,
I am currently building Thunderbird version 91 for aarch64. I noticed that I now need to comment out the line involving ROOT_CLIP_CHAIN in mozilla-esr91 as below:

  • cd ~/mozilla-esr91
  • nano gfx/webrender_bindings/webrender_ffi.h
    Comment out line 80:
  • // const uint64_t ROOT_CLIP_CHAIN = ~0;
    I think the last time I built from mozilla-esr91 was 2 weeks ago. I did not need to make this change back then when building.

Related to bug 1773259, apparently using a newer version of cbingen than before causes the issue.

Thanks Rachel,
I'm leaving my system with the latest cbindgen, and just applying my patch prior to each build. It's an interesting debate: whether to install the updated cbindgen on the official build systems (which would require updating the source in gfx/webrender_bindings/webrender_ffi.h).
All the best,
Daniel

Summary: Building Thunderbird 102 beta on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments June 17 Aus time, June 16 PDT) → Building Thunderbird 102 on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments une 16 PDT)
Summary: Building Thunderbird 102 on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments une 16 PDT) → Building Thunderbird 102 on aarch64 for aarch64 WASM related errors (The best solution I could achieve is in comments June 16 PDT)

Hi,
The method (described above) reliably builds a Thunderbird 102 release version (aarch64 Linux) from the mozilla-esr102 and comm-esr102 repositories.
All the best,
Daniel

Status: UNCONFIRMED → RESOLVED
Closed: 2 years ago
Resolution: --- → DUPLICATE
Status: RESOLVED → UNCONFIRMED
OS: Unspecified → Linux
Hardware: Unspecified → ARM64
Resolution: DUPLICATE → ---

Hi,
The cbindgen (ROOT_CLIP_CHAIN) issue was just a minor item mentioned in this bug report. This bug report is not a duplicate of the cbindgen (ROOT_CLIP_CHAIN) issue. The main issue in my bug report was the WASM related errors that occurred during building Thunderbird 102 for Linux aarch64 on a Linux aarch64 host (i.e. I do not cross-compile). These WASM related erros still occcur when building Thunderbird 102 for Linux aarch64 on a Linux aarch64 host right now. I checked this when I did a build just a few hours ago.
I have also noticed that almost identical WASM related errors occured when I tried building Thunderbird 102 for Windows using MozillaBuild on a Windows 10 host (Nb: The host was a VMware Workstation virtual machine running Windows 10 Pro). i.e. I did not cross-compille. Therefore this bug report is probably relevant to architectures other than Linux aarch64.
The method described above (deleting undefined references from rlbox.wasm.c) allows both the Linux aarch64 and Windows builds described above to complete successfully. I have not tried building with other architectures. The method described above is messy because the build fails, then I delete the offending lines in rlbox.wasm.c, and then I recommence the build (which succeeds the second time around). There should be a way to fix this so the build just succeeds the first time. I am still trying different methods to come up with a solution.
I recently tried:

  1. For aarch64 builds from an aarch64 host: I tried using a wasi-sysroot created with LLVM-14.0.3 and build using LLVM-14.03. This did not resolve the error. I plan to try this in the next few days: wasi-sysroot created with LLVM-14.0.4 and build using LLVM-14.04.
  2. For Windows builds from a Windows host: I plan to try this in the next few days: wasi-sysroot created with LLVM-14.04. Nb: ./mach bootstrap uses LLVM-14.04 with MozillaBiuld on a Windows host.
    All the best,
    Daniel

Sorry about the mis-filing.

Our CI does run a Windows build on Windows hosts targeting x64 at least once daily - but only on comm-central. I just started up one-off on comm-esr102; we shall see how it fares.

Maybe you mentioned this above, but is there a reason why you are not cross compiling? Aarch64 systems are not a supported build host (with the exception of Mac M1s). https://firefox-source-docs.mozilla.org/build/buildsystem/supported-configurations.html#supported-build-hosts

Linux-aarch64 is a Tier 2 cross-compile supported target. Builds are run at least once daily on comm-central (and I personally run that build on a Raspi400).

Hi Rob,
No worries. Thanks for the info, particularly on supported build hosts. I must admit that my first trial of building aarch64 Linux targets was cross-compiling using an Ubuntu host. However, I ran into some problems and ended up successfully building on an aarch64 host. Thunderbird esr91 did not have the rlbox wasm feature and therefore I did not run into any of the problems in this bug report.
All the best,
Daniel

Mind if I close this then?

Hi Rob,
My builds are working well, so I don't mind if you close this report. I'll open a bug if I find that I can't build anymore.
All the best,
Daniel

Just for future reference:
I tried building the latest from mozilla-central and comm-central. I still get the same difficulties.
I tried changing LLVM minor versions (14.0.3, 14.0.4 and 14.0.6). I built the wasi compiler-rt and wasi-sysroot on each occasion with each corresponding respective LLVM minor version. All of this did not make a difference. The wasm related errors still occurred.
I noticed that:
obj-aarch64-unknown-linux-gnu/security/rlbox/rlbox.wasm.h
contains:
extern u32 Z_envZ___cxa_allocate_exceptionZ_ii(void*, u32);
extern void Z_envZ___cxa_throwZ_viii(void*, u32, u32, u32);
extern void Z_envZ___multi3Z_vijjjj(void*, u32, u64, u64, u64, u64);
However, these definitions are not carried through to rlbox.wasm.c despite the include statement.
I tried adding these definitions to rlbox.wasm.c
However, the build did not proceed successfully.

Status: UNCONFIRMED → RESOLVED
Closed: 2 years ago2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: