1522609 - linker can't find LMDB symbols on OS X Cross Compiled asan (build-macosx64-asan-fuzzing/opt)

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Myk Melez [:myk] [@mykmelez]]

The OS X Cross Compiled asan (build-macosx64-asan-fuzzing/opt) build (https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2bf1f1e907ae1ac3ad1341ff1de660fc4739bab&selectedJob=223405495&searchStr=os%2Cx%2Ccross%2Ccompiled%2Casan%2Cbuild-macosx64-asan-fuzzing%2Fopt%2C%28bof%29) is failing on the patch for bug 1490496 because the linker can't find the LMDB C library's symbols:

INFO - Undefined symbols for architecture x86_64:
INFO - "_mdb_cursor_get", referenced from:
INFO - _ZN63$LT$kvstore..task..EnumerateTask$u20$as$u20$moz_task..Task$GT$3run17h8dfc468f071e0164E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN84$LT$core..iter..Map$LT$I$C$$u20$F$GT$$u20$as$u20$core..iter..iterator..Iterator$GT$4next17ha229e95c41313eeaE in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_cursor_open", referenced from:
INFO - _ZN63$LT$kvstore..task..EnumerateTask$u20$as$u20$moz_task..Task$GT$3run17h8dfc468f071e0164E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_strerror", referenced from:
INFO - _ZN57$LT$lmdb..error..Error$u20$as$u20$core..fmt..Display$GT$3fmt17h84518602debea21dE in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_cursor_close", referenced from:
INFO - _ZN63$LT$kvstore..task..EnumerateTask$u20$as$u20$moz_task..Task$GT$3run17h8dfc468f071e0164E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_env_close", referenced from:
INFO - _ZN65$LT$kvstore..task..GetOrCreateTask$u20$as$u20$moz_task..Task$GT$3run17hc5845cb04699e2e1E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN34$LT$alloc..sync..Arc$LT$T$GT$$GT$9drop_slow17h9809c59af9f69d16E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_env_open", referenced from:
INFO - _ZN65$LT$kvstore..task..GetOrCreateTask$u20$as$u20$moz_task..Task$GT$3run17hc5845cb04699e2e1E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_env_create", referenced from:
INFO - _ZN65$LT$kvstore..task..GetOrCreateTask$u20$as$u20$moz_task..Task$GT$3run17hc5845cb04699e2e1E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_txn_abort", referenced from:
INFO - _ZN57$LT$kvstore..task..PutTask$u20$as$u20$moz_task..Task$GT$3run17hd9a4fe9675f4d0d9E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN57$LT$kvstore..task..GetTask$u20$as$u20$moz_task..Task$GT$3run17ha28a6584af23a26bE in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN57$LT$kvstore..task..HasTask$u20$as$u20$moz_task..Task$GT$3run17h4eeaf11cf898a72dE in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN60$LT$kvstore..task..DeleteTask$u20$as$u20$moz_task..Task$GT$3run17hafacd7f285f35bd5E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN63$LT$kvstore..task..EnumerateTask$u20$as$u20$moz_task..Task$GT$3run17h8dfc468f071e0164E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - __ZN4lmdb11environment11Environment9create_db17h36451460944e2b88E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_put", referenced from:
INFO - _ZN57$LT$kvstore..task..PutTask$u20$as$u20$moz_task..Task$GT$3run17hd9a4fe9675f4d0d9E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_txn_commit", referenced from:
INFO - _ZN57$LT$kvstore..task..PutTask$u20$as$u20$moz_task..Task$GT$3run17hd9a4fe9675f4d0d9E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN60$LT$kvstore..task..DeleteTask$u20$as$u20$moz_task..Task$GT$3run17hafacd7f285f35bd5E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - __ZN4lmdb11environment11Environment9create_db17h36451460944e2b88E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_dbi_open", referenced from:
INFO - __ZN4lmdb11environment11Environment9create_db17h36451460944e2b88E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_txn_begin", referenced from:
INFO - _ZN57$LT$kvstore..task..PutTask$u20$as$u20$moz_task..Task$GT$3run17hd9a4fe9675f4d0d9E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN57$LT$kvstore..task..GetTask$u20$as$u20$moz_task..Task$GT$3run17ha28a6584af23a26bE in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN57$LT$kvstore..task..HasTask$u20$as$u20$moz_task..Task$GT$3run17h4eeaf11cf898a72dE in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN60$LT$kvstore..task..DeleteTask$u20$as$u20$moz_task..Task$GT$3run17hafacd7f285f35bd5E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - _ZN63$LT$kvstore..task..EnumerateTask$u20$as$u20$moz_task..Task$GT$3run17h8dfc468f071e0164E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - __ZN4lmdb11environment11Environment9create_db17h36451460944e2b88E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_get", referenced from:
INFO - _ZN56$LT$rkv..readwrite..Reader$LT$$u27$env$C$$u20$K$GT$$GT$3get17h5db47cd8ee354defE in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_env_set_maxdbs", referenced from:
INFO - _ZN65$LT$kvstore..task..GetOrCreateTask$u20$as$u20$moz_task..Task$GT$3run17hc5845cb04699e2e1E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - "_mdb_del", referenced from:
INFO - _ZN60$LT$kvstore..task..DeleteTask$u20$as$u20$moz_task..Task$GT$3run17hafacd7f285f35bd5E in libgkrust.a(gkrust-8c57c405dd4cdd98.gkrust.9vncoaew-cgu.0.rcgu.o)
INFO - ld: symbol(s) not found for architecture x86_64
INFO - clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
INFO - /builds/worker/workspace/build/src/config/rules.mk:689: recipe for target 'XUL' failed
ERROR - make[4]: *** [XUL] Error 1
INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library'
INFO - /builds/worker/workspace/build/src/config/recurse.mk:74: recipe for target 'toolkit/library/target' failed
ERROR - make[3]: *** [toolkit/library/target] Error 2
INFO - /builds/worker/workspace/build/src/config/recurse.mk:32: recipe for target 'compile' failed
ERROR - make[2]: *** [compile] Error 2
INFO - /builds/worker/workspace/build/src/config/rules.mk:415: recipe for target 'default' failed
ERROR - make[1]: *** [default] Error 2
INFO - client.mk:125: recipe for target 'build' failed
INFO - make: *** [build] Error 2

Comparing the log for that build (https://taskcluster-artifacts.net/HajrffwzRsG1y8qWnkW4tw/0/public/logs/live_backing.log) against the log for an OS X Cross Compiled opt (build-macosx64/opt) build (https://taskcluster-artifacts.net/dX3Jj5huQhyU4vVRYmLzfA/0/public/logs/live_backing.log), which succeeds, shows that they have very different build-related environment variables and build command invocations:

build-macosx64-asan-fuzzing/opt (https://taskcluster-artifacts.net/HajrffwzRsG1y8qWnkW4tw/0/public/logs/live_backing.log):

INFO - [lmdb-sys 0.8.0] CC_x86_64_apple_darwin = None
INFO - [lmdb-sys 0.8.0] TARGET_CC = None
INFO - [lmdb-sys 0.8.0] CC = None
…
INFO - [lmdb-sys 0.8.0] CFLAGS_x86_64_apple_darwin = None
…
INFO - [lmdb-sys 0.8.0] running: "cc" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-Wall" "-Wextra" "-o" "/builds/worker/workspace/build/src/obj-firefox/x86_64-apple-darwin/release/build/lmdb-sys-6f08de174e0c3cd5/out/mdb.o" "-c" "/builds/worker/workspace/build/src/third_party/rust/lmdb-sys/lmdb/libraries/liblmdb/mdb.c"

build-macosx64/opt (https://taskcluster-artifacts.net/dX3Jj5huQhyU4vVRYmLzfA/0/public/logs/live_backing.log):

INFO - [lmdb-sys 0.8.0] CC_x86_64_apple_darwin = Some(" /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang -isysroot /builds/worker/workspace/build/src/MacOSX10.11.sdk -std=gnu99 --target=x86_64-darwin11")
…
INFO - [lmdb-sys 0.8.0] CFLAGS_x86_64_apple_darwin = Some("-fvisibility=hidden -fvisibility-inlines-hidden -DNDEBUG=1 -DTRIMMED=1 -I/builds/worker/workspace/build/src/toolkit/library/rust -I/builds/worker/workspace/build/src/obj-firefox/toolkit/library/rust -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread -pipe -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O3 -fno-omit-frame-pointer -funwind-tables")
…
INFO - [lmdb-sys 0.8.0] running: "/builds/worker/workspace/build/src/sccache2/sccache" "/builds/worker/workspace/build/src/clang/bin/clang" "-isysroot" "/builds/worker/workspace/build/src/MacOSX10.11.sdk" "-std=gnu99" "--target=x86_64-darwin11" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-fvisibility=hidden" "-fvisibility-inlines-hidden" "-DNDEBUG=1" "-DTRIMMED=1" "-I/builds/worker/workspace/build/src/toolkit/library/rust" "-I/builds/worker/workspace/build/src/obj-firefox/toolkit/library/rust" "-I/builds/worker/workspace/build/src/obj-firefox/dist/include" "-I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr" "-I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss" "-fPIC" "-include" "/builds/worker/workspace/build/src/obj-firefox/mozilla-config.h" "-DMOZILLA_CLIENT" "-Qunused-arguments" "-U_FORTIFY_SOURCE" "-D_FORTIFY_SOURCE=2" "-fstack-protector-strong" "-U_FORTIFY_SOURCE" "-D_FORTIFY_SOURCE=2" "-fstack-protector-strong" "-fno-strict-aliasing" "-ffunction-sections" "-fdata-sections" "-fno-math-errno" "-pthread" "-pipe" "-g" "-Xclang" "-load" "-Xclang" "/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so" "-Xclang" "-add-plugin" "-Xclang" "moz-check" "-O3" "-fno-omit-frame-pointer" "-funwind-tables" "--target=x86_64-apple-darwin" "-o" "/builds/worker/workspace/build/src/obj-firefox/x86_64-apple-darwin/release/build/lmdb-sys-6f08de174e0c3cd5/out/mdb.o" "-c" "/builds/worker/workspace/build/src/third_party/rust/lmdb-sys/lmdb/libraries/liblmdb/mdb.c"

Perhaps this is because the part of rules.mk that "pass(es) the compilers and flags in use to cargo for use in build scripts" notes that we "don't do this for ASAN/TSAN builds because we don't pass our custom linker (see below) which will muck things up" (https://searchfox.org/mozilla-central/rev/c035ee7d3a5cd6913e7143e1bce549ffb4a566ff/config/rules.mk#901-903).

It looks like geckodriver had a similar problem in bug 1417646 and worked around it by disabling ASAN builds, which Ted explains in more detail in bug 1417646, comment 84. However, in this case we can't simply disable the feature in ASAN builds, because we can't easily disable the features that will depend on it.

Comment 1

[(not currently active) Ted Mielczarek]

Frustratingly I still don't have a good answer for you here. ☹️ cargo does not provide a way to distinguish between "the compiler/linker to use for target binaries" and "the compiler/linker to use for host binaries" when the host and target are the same platform.

Comment 2

[Kim Moir [:kmoir] ET]

:glandium ted mentioned that you might have some insight on this problem

Comment 3

[Mike Hommey [:glandium]]

I believe the current queue of things that I'm working on to, among others, fix bug 1522614 will fix this.

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1522609 - Pass HOST_CC/HOST_CXX/HOST_CFLAGS/HOST_CXXFLAGS down to cargo.

While this isn't related to the bug, since we're going to touch the
cargo compiler flags, we might as well do this too.

It wasn't previously reliable to pass those flags down because what
cargo uses as target for build scripts and procedural macros is
determined by the rust host, which was not necessarily the same as the
build system host. But as of bug 1523143, they are always the same.

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1522609 - Pass compilers and flags down to cargo on sanitizer/fuzzying/ccov builds.

Depends on D18280

Comment 6

[Mike Hommey [:glandium]]

(In reply to Myk Melez [:myk] [@mykmelez] from comment #0)

The OS X Cross Compiled asan (build-macosx64-asan-fuzzing/opt) build (https://treeherder.mozilla.org/#/jobs?repo=try&revision=c2bf1f1e907ae1ac3ad1341ff1de660fc4739bab&selectedJob=223405495&searchStr=os%2Cx%2Ccross%2Ccompiled%2Casan%2Cbuild-macosx64-asan-fuzzing%2Fopt%2C%28bof%29) is failing on the patch for bug 1490496 because the linker can't find the LMDB C library's symbols:

I got green Windows aarch64 and OSX asan fuzzing builds with that patch + the stack from bug 1522614 and this one.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e1d8da551f52
Pass HOST_CC/HOST_CXX/HOST_CFLAGS/HOST_CXXFLAGS down to cargo. r=ted
https://hg.mozilla.org/integration/autoland/rev/17ee66cc26ef
Pass compilers and flags down to cargo on sanitizer/fuzzying/ccov builds. r=ted

Comment 8

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/e1d8da551f52
https://hg.mozilla.org/mozilla-central/rev/17ee66cc26ef