Closed Bug 1723245 Opened 3 years ago Closed 3 years ago

Fuzzing build fails due to undefined sanitizer symbols (__start___sancov_cntrs) when instantiating template in header file

Categories

(Firefox Build System :: General, defect, P3)

Product:
Firefox Build System
Component:
General
Type:
defect

Tracking

(firefox93 fixed)

Status:
RESOLVED FIXED
Milestone:
93 Branch
Tracking Flags:
Tracking Status
firefox93 --- fixed

People

(Reporter: sfink, Assigned: glandium)

References

Details

Attachments

(2 files)

verbose-compiler-and-linker-output.txt
279.65 KB, text/plain
Details
Bug 1723245 - Disable --gc-sections when building with sanitizer coverage.
48 bytes, text/x-phabricator-request
Details | Review

Try server revision ea8a407a0f9a or its parent fb639ed5a772 on the SM(fuzzing) job:

[task 2021-07-29T17:56:45.957Z]  3:59.27 js/src/build/libmozjs-92a1.so
[task 2021-07-29T17:56:54.452Z]  4:07.76 /usr/bin/ld: ../StructuredClone.o: in function `sancov.module_ctor_8bit_counters':
[task 2021-07-29T17:56:54.453Z]  4:07.76 StructuredClone.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0x4): undefined reference to `__start___sancov_cntrs'
[task 2021-07-29T17:56:54.454Z]  4:07.77 /usr/bin/ld: StructuredClone.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0xb): undefined reference to `__stop___sancov_cntrs'
[task 2021-07-29T17:56:54.455Z]  4:07.77 /usr/bin/ld: StructuredClone.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0x17): undefined reference to `__start___sancov_pcs'
[task 2021-07-29T17:56:54.456Z]  4:07.77 /usr/bin/ld: StructuredClone.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0x1e): undefined reference to `__stop___sancov_pcs'

(followed by lots of /usr/bin/ld: DWARF error: ... errors).

ea8a407a0f9a is the try push rev. Parent fb639ed5a772 has the problematic changes. Grandparent 0896ae409c2c is fine.

I have not managed to reproduce in my local dev environment, but I can reproduce it with a docker image matching that job.

The failure happens while linking libmozjs-92a1.so. In either the good or bad revisions, 37 of the object files that go into it have the same undefined symbols:

                 U __start___sancov_cntrs
                 U __start___sancov_pcs
                 U __stop___sancov_cntrs
                 U __stop___sancov_pcs

In the good revision, the successful libmozjs-92a1.so contains:

0000000004d1f490 d __start___sancov_cntrs
0000000004d57850 d __start___sancov_pcs
0000000004d5784f d __stop___sancov_cntrs
00000000050486f0 d __stop___sancov_pcs

The command line is:

/builds/worker/fetches/clang/bin/clang++ --sysroot /builds/worker/fetches/sysroot-x86_64-linux-gnu -std=gnu++17 -U_FORTIFY_SOURCE -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wdeprecated-this-capture -Wempty-body -Wformat-type-confusion -Wignored-qualifiers -Wpointer-arith -Wshadow-field-in-constructor-modified -Wshadow-uncaptured-local -Wsign-compare -Wtype-limits -Wno-error=tautological-type-limit-compare -Wunreachable-code -Wunreachable-code-return -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wno-range-loop-analysis -Wc++2a-compat -Wcomma -Wenum-compare-conditional -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=free-nonheap-object -Wno-error=return-std-move -Wno-error=atomic-alignment -Wno-error=deprecated-copy -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-psabi -Wno-unknown-warning-option -fno-sized-deallocation -fno-aligned-new -fsanitize=address -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -gline-tables-only -gdwarf-2 -Xclang -load -Xclang /builds/worker/checkouts/gecko/obj-spider/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O3 -fno-omit-frame-pointer -funwind-tables -Werror  -shared -Wl,--gc-sections -Wl,-h,libmozjs-92a1.so -o libmozjs-92a1.so /builds/worker/checkouts/gecko/obj-spider/js/src/build/libmozjs-92a1_so.list   -lpthread -fsanitize=fuzzer-no-link -rdynamic -fsanitize=address -rdynamic -Wl,-rpath-link,/builds/worker/fetches/sysroot-x86_64-linux-gnu/lib/x86_64-linux-gnu -Wl,-rpath-link,/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -Wl,-Bsymbolic -Wl,-rpath-link,/builds/worker/checkouts/gecko/obj-spider/dist/bin -Wl,-rpath-link,/builds/worker/checkouts/gecko/obj-spider/dist/lib  /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a ../../../config/external/nspr/pr/libnspr4.so ../../../config/external/nspr/libc/libplc4.so ../../../config/external/nspr/ds/libplds4.so -Wl,--version-script,symverscript  -lm  -lz -lm -ldl/builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a

The source code changes add some templatized classes to some header files, implement a few of their methods, and update macro uses in a couple of files.

I'm not too clear on how this is supposed to work, but for most sanitizer symbols I thought -fsanitizer=address was supposed to make the compile generate references to various sanitizer symbols, and the same flag for the linker made it link with sanitizer libraries. But the above symbols don't seem to be provided by the sanitizer libs. They are something more internally generated? I think they're section names. I would still guess that -fsanitizer=address is supposed to somehow provide them.

If I copy $objdir/js/src/*.o from a "bad" $objdir to a good one, it breaks (in the same way as above). Copying the good files back fixes it. In both good and bad objdirs, there are 37 .o files with the undefined symbols. So it's linking together a bunch of object files with those symbols undefined, with the same command line, yet it's providing the definitions in one case and not the other.

I'm sure this makes somehow.

The last piece I don't understand is that this is a link to generate a shared library. The docs say "The AddressSanitizer run-time library should be linked to the final executable, so make sure to use clang (not ld) for the final link step. When linking shared libraries, the AddressSanitizer run-time is not linked, so -Wl,-z,defs may cause link errors (don’t use it with AddressSanitizer)." I'm not totally clear on why it needs these to be defined in the first place for a shared library. Also, the errors are coming from ld:

/builds/worker/fetches/clang/bin/clang++ --sysroot /builds/worker/fetches/sysroot-x86_64-linux-gnu -std=gnu++17 -U_FORTIFY_SOURCE -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wdeprecated-this-capture -Wempty-body -Wformat-type-confusion -Wignored-qualifiers -Wpointer-arith -Wshadow-field-in-constructor-modified -Wshadow-uncaptured-local -Wsign-compare -Wtype-limits -Wno-error=tautological-type-limit-compare -Wunreachable-code -Wunreachable-code-return -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wno-range-loop-analysis -Wc++2a-compat -Wcomma -Wenum-compare-conditional -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=free-nonheap-object -Wno-error=return-std-move -Wno-error=atomic-alignment -Wno-error=deprecated-copy -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-psabi -Wno-unknown-warning-option -fno-sized-deallocation -fno-aligned-new -fsanitize=address -fPIC -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -gline-tables-only -gdwarf-2 -Xclang -load -Xclang /builds/worker/checkouts/gecko/obj-spider/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O3 -fno-omit-frame-pointer -funwind-tables -Werror  -shared -Wl,--gc-sections -Wl,-h,libmozjs-92a1.so -o libmozjs-92a1.so /builds/worker/checkouts/gecko/obj-spider/js/src/build/libmozjs-92a1_so.list   -lpthread -fsanitize=fuzzer-no-link -rdynamic -fsanitize=address -rdynamic -Wl,-rpath-link,/builds/worker/fetches/sysroot-x86_64-linux-gnu/lib/x86_64-linux-gnu -Wl,-rpath-link,/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -Wl,-Bsymbolic -Wl,-rpath-link,/builds/worker/checkouts/gecko/obj-spider/dist/bin -Wl,-rpath-link,/builds/worker/checkouts/gecko/obj-spider/dist/lib  /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a ../../../config/external/nspr/pr/libnspr4.so ../../../config/external/nspr/libc/libplc4.so ../../../config/external/nspr/ds/libplds4.so -Wl,--version-script,symverscript  -lm  -lz -lm -ldl
/usr/bin/ld: ../StructuredClone.o: in function `sancov.module_ctor_8bit_counters':
StructuredClone.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0x4): undefined reference to `__start___sancov_cntrs'
/usr/bin/ld: StructuredClone.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0xb): undefined reference to `__stop___sancov_cntrs'
/usr/bin/ld: StructuredClone.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0x17): undefined reference to `__start___sancov_pcs'
/usr/bin/ld: StructuredClone.cpp:(.text.sancov.module_ctor_8bit_counters[sancov.module_ctor_8bit_counters]+0x1e): undefined reference to `__stop___sancov_pcs'
/usr/bin/ld: /usr/bin/ld: DWARF error: mangled line number section (bad file number)
/usr/bin/ld: DWARF error: mangled line number section (bad file number)
/usr/bin/ld: DWARF error: mangled line number section (bad file number)
/usr/bin/ld: DWARF error: mangled line number section (bad file number)
/usr/bin/ld: DWARF error: mangled line number section (bad file number)

which makes me wonder if it should not be using ld here?

I'm just throwing out random comments at this point. I'm lost.

The linker command in both cases, from adding -v:

"/usr/bin/ld" --sysroot=/builds/worker/fetches/sysroot-x86_64-linux-gnu -export-dynamic --hash-style=both --eh-frame-hdr -m elf_x86_64 -shared -export-dynamic -o libmozjs-92a1.so /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crti.o /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/crtbeginS.o -L/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0 -L/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu -L/builds/worker/fetches/sysroot-x86_64-linux-gnu/lib/x86_64-linux-gnu -L/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu -L/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../.. -L/builds/worker/fetches/sysroot-x86_64-linux-gnu/lib -L/builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib --gc-sections -h libmozjs-92a1.so /builds/worker/checkouts/gecko/obj-spider/js/src/build/libmozjs-92a1_so.list -lpthread -rpath-link /builds/worker/fetches/sysroot-x86_64-linux-gnu/lib/x86_64-linux-gnu -rpath-link /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/x86_64-linux-gnu -z noexecstack -z text -z relro -z nocopyreloc -Bsymbolic-functions --build-id=sha1 -Bsymbolic -rpath-link /builds/worker/checkouts/gecko/obj-spider/dist/bin -rpath-link /builds/worker/checkouts/gecko/obj-spider/dist/lib /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a ../../../config/external/nspr/pr/libnspr4.so ../../../config/external/nspr/libc/libplc4.so ../../../config/external/nspr/ds/libplds4.so --version-script symverscript -lm -lz -lm -ldl -lstdc++ -lm -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/crtendS.o /builds/worker/fetches/sysroot-x86_64-linux-gnu/usr/lib/gcc/x86_64-linux-gnu/7.5.0/../../../x86_64-linux-gnu/crtn.o

I will also note that if I combine the .o files from a "good" build and a "bad" build, I get:

  • if StructuredClone.o is from the good build, it will always link fine
  • if StructuredClone.o is from the bad build, it will usually fail, but depending on where the other .o files come from, it will succeed 8% of the time (64 out of 772 runs with files randomly chosen from good and bad builds)
  • in the successful links, all of the following files have to come from the good build. If any of them comes from the bad build, it will fail:
    • CTypes.o
    • Interpreter.o
    • RegExp.o
    • jsmath.o

(no other .o files seem to matter.)

The patch does not touch StructuredClone.cpp, but it does modify the js/public/experimental/TypedData.h header that it includes.

Blocks: 1720422

Hey, sorry, this slipped through the cracks a bit.
It could be worth bisecting the revision causing the problem (fb639ed5a772d4eb69fbf1e58933ee2b77cf2ff7?) to see which specific changes are causing the issue. I'll dig into this a bit before you come online today.
It's surprising that you aren't able to reproduce this in a local dev environment - perhaps you're using a slightly different version of clang?

Yes, yesterday I started chopping up the patch. I have it down to a patch that introduces a templatized class, including specializations for a few methods. There are no calls to the constructors or any other methods of the classes. All possible specializations (there's a list of valid types) are explicitly instantiated. Maybe I'll try removing that too and see what happens.

Sweet, sounds like you're further along than I am.
Please NI me with a link to the try revision once you've got a minimal reproduce, if that works for you 👍

Yes, it starts happening as soon as I explicitly instantiate the templates. Which I'm a little surprised I'm doing in the header file; I think a later patch in the stack moves the instantiation to a .cpp file. Anyway, I have a patch stack where this patch changes it from linking successfully to failing with the ___sancov errors:

diff --git a/js/public/experimental/TypedData.h b/js/public/experimental/TypedData.h
--- a/js/public/experimental/TypedData.h
+++ b/js/public/experimental/TypedData.h
@@ -610,6 +610,11 @@ namespace JS {
 JS_FOR_EACH_TYPED_ARRAY(IMPL_TYPED_ARRAY_CLASS)
 #undef IMPL_TYPED_ARRAY_CLASS
 
+#define JS_DECLARE_CLASS_PTR(ExternalType, NativeType, Name) \
+  template class TypedArray<js::Scalar::Name>;
+JS_FOR_EACH_TYPED_ARRAY(JS_DECLARE_CLASS_PTR)
+#undef JS_DECLARE_CLASS_PTR
+
 }  // namespace JS
 
 #endif  // js_experimental_TypedData_h

As for clang versions, it's 12.0.1 both places.

FWIW, if you move the instantiation to a non-header file, does linking work again?

I'll note that in the docker container, I reproduce this with either the "real" command (launched via run-task) or going into a checkout and running js/src/devtools/automation/autospider.py fuzzing. I still cannot reproduce locally.

I did notice one difference in the ld command lines between local and docker: docker runs with ~/fetches/clang/bin/ld.lld while locally I get /usr/bin/ld (which is ld.bfd). Switching to ld.lld locally still works. Switching docker to /usr/bin/ld (which is a symlink to x86_64-linux-gnu-ld; I'm assuming that's ld.bfd?) still fails, with the output

ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN11encoding_rs7variant14VariantEncoder17has_pending_state17h3f259934fe798f2aE)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN11encoding_rs8Encoding16utf8_valid_up_to17hf20f90991a3a3238E)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN11encoding_rs7Encoder45max_buffer_length_from_utf8_if_no_unmappables17hd0541b46c8c5e71cE)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN11encoding_rs7Encoder36encode_from_utf8_without_replacement17h4716962a37af537fE)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN11encoding_rs7Encoder46max_buffer_length_from_utf16_if_no_unmappables17h95aa4e6ca2e6c2beE)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN11encoding_rs7Encoder37encode_from_utf16_without_replacement17h69373a7ba705b800E)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN10wasmparser6parser6Parser3new17h6263fda68619e8e9E)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN10wasmparser6parser6Parser9parse_all17h647f5279bc190bc7E)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN109_$LT$wasmparser..readers..operators..OperatorsReader$u20$as$u20$core..iter..traits..collect..IntoIterator$GT$9into_iter17h3b77b2db16454cf6E)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN4wast6parser11ParseBuffer6parser17hb6a436d26827f858E)
ld.lld: error: /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(__sancov_cntrs): sh_link points to discarded section /builds/worker/checkouts/gecko/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o):(.text._ZN4wast6parser6Parser6cursor17hb29af7a168cffa8dE)

and if I remove the --gc-sections argument from the command line, it works! Odd that the above error doesn't mention StructuredClone.o anywhere.

Ok, my try pushes are ready. I have a working push followed by a failing push with a single additional patch (which is the above patch, but with s/JS_DECLARE_CLASS_PTR/INSTANTIATE/ to better reflect what it's doing.)

Flags: needinfo?(mhentges)

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #9)

FWIW, if you move the instantiation to a non-header file, does linking work again?

Yes, it does!

Ok, next thing is to make that change and see if the problem reappears when I rebase the rest of my patch stack on top of it.

Assignee: nobody → mhentges
Status: NEW → ASSIGNED
Flags: needinfo?(mhentges)

Hopefully the rebased stack will be happy with the instantiation in a non-header file 👍.

I'd still like to investigate to see why the in-header instantiation was a problem. I'm currently working to reproduce the problem in a local docker image, then ideally on my host system soon enough. I'll write my findings here once I've got some concrete answers.

I've reproduced a similar form of a failure locally, outside of a Docker container. I did so by:

  1. Checking out the problematic revision (a6b43e1) and pushing a fresh task to repro the problem in CI
  2. Downloading all the MOZ_FETCHES defined in the task, extracting them, and putting them in ~/.mozbuild
    • Note: I had to rustup self uninstall to make sure that I used the fetch'd Rust version
  3. I defined mozconfig.autospider:
ac_add_options --enable-project=js
ac_add_options --enable-stdcxx-compat --disable-gold --enable-fuzzing --enable-gczeal --enable-debug-symbols='-gline-tables-only -gdwarf-2' --disable-jemalloc --disable-stdcxx-compat --enable-address-sanitizer --enable-ctypes --enable-nspr-build --enable-rust-simd --enable-clang-plugin --enable-optimize --disable-debug --enable-nspr-build
  1. MOZ_AUTOMATION=1 AUTOMATION=1 SPIDERMONKEY_VARIANT="fuzzing" MOZCONFIG=/home/mitch/dev/firefox/mozconfig.autospider ./mach build

Here's my build failure output:

<snip>
js/src/build/libjs_static.a
js/src/build/libmozjs-92a1.so
`__sancov_cntrs' referenced in section `.text.wasm_code_offsets[wasm_code_offsets]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text.wasm_code_offsets[wasm_code_offsets]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text.wasm_code_offsets[wasm_code_offsets]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text.encoder_encode_from_utf16_without_replacement[encoder_encode_from_utf16_without_replacement]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text.encoder_max_buffer_length_from_utf16_if_no_unmappables[encoder_max_buffer_length_from_utf16_if_no_unmappables]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text.encoder_encode_from_utf8_without_replacement[encoder_encode_from_utf8_without_replacement]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text.encoder_max_buffer_length_from_utf8_if_no_unmappables[encoder_max_buffer_length_from_utf8_if_no_unmappables]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text.encoder_has_pending_state[encoder_has_pending_state]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text.encoding_utf8_valid_up_to[encoding_utf8_valid_up_to]' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text._ZN3wat10_parse_str17hc00a40d1121f720bE' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text._ZN3wat10_parse_str17hc00a40d1121f720bE' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
`__sancov_cntrs' referenced in section `.text._ZN3wat10_parse_str17hc00a40d1121f720bE' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /home/mitch/dev/firefox/obj-x86_64-pc-linux-gnu/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-306a8347296095f4.jsrust.3vmxjjt6-cgu.0.rcgu.o)
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
<snip>

Rebased stack is looking good, so it appears I'm no longer blocked on this.

No longer blocks: 1720422

Information dump/current status:

Is this a linker issue?

As noted in this comment, one theory is that we were incorrectly using ld (ld.bfd) instead of LLVM's lld.
However, looking into the referenced docs again, I'm realizing that it's saying that we need to let clang manage the linking, rather than us doing the linking ourselves.
Indeed, clang is managing the linking when it fails - this is clear, because underneath the link errors, clang says that it was calling the linker: clang-12: error: linker command failed ....

What is the cause of this issue?

I'm not sure yet - one theory is that this is a Clang bug that's related to instantiating templates in header files and address sanitization.
Additional work is needed to narrow down the cause, though - probably whittling down the build into a minimal reproduce, which is a non-trivial amount of work.

Can the issue be reproduced in a local dev environment?

Yes, see this comment

Next steps

Since this is work-aroundable at the moment, I'm leaning towards deferring this - perhaps it'll be encountered and resolved upstream.
Unless someone knows a clear cause/reason behind this failure, debugging it will take me multiple days, which I'm not sure is worth it just yet.
I'll re-title this bug to better capture its current state and send it to the backlog.

If we do encounter this issue again, at least we have some useful information logged in this ticket which can help in the future.
Thanks for your help digging into this Steve :)

Summary: undefined sanitizer symbols (eg __start___sancov_cntrs) after seemingly irrelevant change → Fuzzing build fails due to undefined sanitizer symbols (__start___sancov_cntrs) when instantiating template in header file
Assignee: mhentges → nobody
Status: ASSIGNED → NEW
Priority: -- → P3

add --verbose -Wl,--verbose to the command line. Chances are it's saying the runtime is missing
~glandium

Glandium had a potential theory in Matrix, but I've got the associated log here and unfortunately it isn't complaining about a missing runtime.

Newer versions of bfd ld make it slightly clearer what's going wrong:

[task 2021-08-18T07:17:08.822Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.wasm_code_offsets[wasm_code_offsets]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.822Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.wasm_code_offsets[wasm_code_offsets]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.wasm_code_offsets[wasm_code_offsets]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.encoder_encode_from_utf16_without_replacement[encoder_encode_from_utf16_without_replacement]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.encoder_max_buffer_length_from_utf16_if_no_unmappables[encoder_max_buffer_length_from_utf16_if_no_unmappables]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.encoder_encode_from_utf8_without_replacement[encoder_encode_from_utf8_without_replacement]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.encoder_max_buffer_length_from_utf8_if_no_unmappables[encoder_max_buffer_length_from_utf8_if_no_unmappables]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.encoder_has_pending_state[encoder_has_pending_state]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text.encoding_utf8_valid_up_to[encoding_utf8_valid_up_to]' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text._ZN3wat10_parse_str17h83c42b9b31236958E' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text._ZN3wat10_parse_str17h83c42b9b31236958E' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 `__sancov_cntrs' referenced in section `.text._ZN3wat10_parse_str17h83c42b9b31236958E' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o): defined in discarded section `__sancov_cntrs' of /builds/worker/workspace/obj-spider/x86_64-unknown-linux-gnu/release/libjsrust.a(jsrust-152298d23a88fc64.jsrust.19ecom06-cgu.0.rcgu.o)
[task 2021-08-18T07:17:08.823Z]  4:16.46 clang-12: error: linker command failed with exit code 1 (use -v to see invocation)

The "discarded section" part is due to --gc-sections.

Blocks: 1726117
Assignee: nobody → mh+mozilla
Status: NEW → ASSIGNED
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/087654c2acc7
Disable --gc-sections when building with sanitizer coverage. r=firefox-build-system-reviewers,mhentges

https://hg.mozilla.org/mozilla-central/rev/087654c2acc7

Status: ASSIGNED → RESOLVED
Closed: 3 years ago
status-firefox93: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 93 Branch
Regressions: 1726544
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: