2010899 - link time failure on aarch64-apple-darwin rust code under PGO builds using MOZ_LTO=cross

Status: NEW

(Firefox Build System :: Toolchains, defect)

Description

[:gerard-majax]

Working on getting "shippable" builds on TaskCluster for Enterprise, I hit a linker issue:

[task 2026-01-14T22:34:25.517+00:00] 22:34:25     INFO -  /builds/worker/fetches/clang/bin/clang++ -isysroot /builds/worker/fetches/MacOSX26.2.sdk -mmacosx-version-min=11.0 -stdlib=libc++ -std=gnu++20 --target=aarch64-apple-darwin -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fstrict-flex-arrays=1 -fno-rtti -pthread -fno-sized-deallocation -fno-aligned-new -ffunction-sections -fdata-sections -fno-math-errno -fno-exceptions -fPIC -fcrash-diagnostics-dir=/builds/worker/artifacts -mcpu=apple-m1 -gdwarf-4 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O3 -fno-omit-frame-pointer -funwind-tables -Werror  -o ../../../dist/bin/XUL -Wl,@/builds/worker/workspace/obj-build/toolkit/library/build/XUL.list   -flto=thin -Wl,-mllvm,-import-instr-limit=10 -Wl,-mllvm,-import-hot-multiplier=30 -fuse-ld=ld -fstack-protector-strong -lresolv -Wl,-rpath,@executable_path/../Frameworks/ChannelPrefs.framework -Wl,-dead_strip -Wl,-order_file /builds/worker/workspace/obj-build/orderfile.txt -Wl,-object_path_lto,XUL.lto.o/ ../../../js/src/build/libjs_static.a ../../../build/pure_virtual/libpure_virtual.a ../../../aarch64-apple-darwin/release/libgkrust.a ../../../dist/bin/libnss3.dylib ../../../dist/bin/libgkcodecs.dylib ../../../dist/bin/liblgpllibs.dylib ../../../dist/bin/ChannelPrefs ../../../dist/bin/libmozglue.dylib -Wl,-exported_symbols_list,XUL.symbols -dynamiclib -install_name @rpath/XUL -compatibility_version 1 -current_version 1  -framework AuthenticationServices -lbsm -framework IOSurface -framework IOKit -framework AudioToolbox -framework CoreMedia -framework VideoToolbox -Wl,-U,_VTRegisterSupplementalVideoDecoderIfAvailable -framework Accessibility -framework Vision -framework CoreFoundation -framework LocalAuthentication -framework Security -lm -framework AVFoundation -framework CoreVideo -framework Foundation -framework MediaPlayer -framework Metal -framework QuartzCore -framework AddressBook -framework AppKit -framework ApplicationServices -framework AudioUnit -framework Carbon -framework CoreAudio -framework CoreLocation -framework CoreMIDI -framework CoreServices -framework OpenGL -framework ServiceManagement -framework SystemConfiguration -weak_framework ScreenCaptureKit -lcups
[task 2026-01-14T22:34:25.517+00:00] 22:34:25     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/toolkit/library/build'
[task 2026-01-14T22:34:25.517+00:00] 22:34:25     INFO -  Undefined symbols for architecture arm64:
[task 2026-01-14T22:34:25.517+00:00] 22:34:25     INFO -    "__ZN17compiler_builtins3int19specialized_div_rem12u128_div_rem17hce7c1f2c80ca0d09E", referenced from:
[task 2026-01-14T22:34:25.517+00:00] 22:34:25     INFO -        ___divti3 in libgkrust.a(compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.086.rcgu.o)
[task 2026-01-14T22:34:25.517+00:00] 22:34:25     INFO -  ld: symbol(s) not found for architecture arm64
[task 2026-01-14T22:34:25.518+00:00] 22:34:25    ERROR -  clang++: error: linker command failed with exit code 1 (use -v to see invocation)
[task 2026-01-14T22:34:25.518+00:00] 22:34:25    ERROR -  gmake[4]: *** [/builds/worker/checkouts/gecko/config/rules.mk:533: ../../../dist/bin/XUL] Error 1

This proved to be happening only on macOS cross-builds with PGO enabled, on TaskCluster. At first I failed to reproduce locally until I could make use of MOZ_LTO=cross locally.

Digging into it, I found that the culprit was some rust code we have enabled only on --enable-enterprise builds. Namely, we had (I'm planning to remove it as a way to fix the issue) a use of the cookie crate to transfer some, and the transfer was happening after serializing the cookie as a string.

The following code would be enough to trigger the issue

use cookie;
let cookie_str = cookie::Cookie::new("name", "value");
println!("{}", cookie_str.to_string());

This code placed either in our crate or in another one (glean for example) would trigger the issue. I have not been able to identify a specific field leading to the broken state.

I've verified the compiler_builtins from our rustc as well as rustup one (I think they should be the same anyway), over several versions and always found the same (exploring libcompiler_builtins-d4cfd55d1b455f62.rlib):

alex@portable-alex:~/codaz/Mozilla/tmp/rustc/1.90/rust-std-1.90.0-aarch64-apple-darwin/rust-std-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-d4cfd55d1b455f62$ grep u128_div_rem *.C.txt
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.006.rcgu.o.C.txt:0000000000000000 g     F __TEXT,__text .hidden compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.015.rcgu.o.C.txt:0000000000000000         *UND* compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.038.rcgu.o.C.txt:0000000000000000         *UND* compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.086.rcgu.o.C.txt:0000000000000000         *UND* compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.106.rcgu.o.C.txt:0000000000000000         *UND* compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.116.rcgu.o.C.txt:0000000000000000         *UND* compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.124.rcgu.o.C.txt:0000000000000000         *UND* compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.133.rcgu.o.C.txt:0000000000000000         *UND* compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09
compiler_builtins-d4cfd55d1b455f62.compiler_builtins.fd478fddefe92704-cgu.134.rcgu.o.C.txt:0000000000000000         *UND* compiler_builtins::int::specialized_div_rem::u128_div_rem::hce7c1f2c80ca0d09

I am not sure if it is expected that the symbol is defined at once place (first place according to the rcgu number) and undefined elsewhere.