1464084 - libstdc++ compat for std::thread::_M_start_thread is busted

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect)

Description

[Nathan Froyd [:froydnj]]

padenot noticed that he was getting crashes with deep stacks when using std::thread:

(gdb) up 100000
#100060 0x000000000041824b in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)()) ()
(gdb) up
#100061 0x00007ffff76ececd in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) up
#100062 0x000000000041824b in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)()) ()
(gdb) up
#100063 0x00007ffff76ececd in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) up
#100064 0x000000000041824b in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>, void (*)()) ()
(gdb) up
#100065 0x00007ffff76ececd in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

...and this continues on for quite some time.

The problem here is that we override _M_start_thread(shared_ptr, void (*)()):

https://searchfox.org/mozilla-central/source/build/unix/stdc++compat/stdc++compat.cpp#128

But the libstdc++ 6 sources use that symbol to actually start the thread:

https://github.com/gcc-mirror/gcc/blob/gcc-6-branch/libstdc%2B%2B-v3/src/c%2B%2B11/thread.cc#L184-L197

and our override calls _M_start_thread(shared_ptr), which just calls the other overload:

https://github.com/gcc-mirror/gcc/blob/gcc-6-branch/libstdc%2B%2B-v3/src/c%2B%2B11/thread.cc#L170-L182

So we get the infinite recursion seen in padenot's debugging session.

The setup is the same for the libstdc++ 5 sources:

https://github.com/gcc-mirror/gcc/blob/gcc-5-branch/libstdc%2B%2B-v3/src/c%2B%2B11/thread.cc#L129-L155

The libstdc++ 4.9 sources, however, start the thread from the _M_start_thread(shared_ptr) overload:

https://github.com/gcc-mirror/gcc/blob/gcc-4_9-branch/libstdc++-v3/src/c++11/thread.cc#L129-L148

This also means that any other std::thread usages are busted, but at least according to crash-stats, padenot is the first person to create std::thread in anger.  This could silently bite us in other ways if webrtc or skia or something starts using std::thread; presumably our test suite would catch such things, but it would also not totally surprise me if such things made it into Nightly, at least, with minimal testing.

I think this means we have a problem, because we need to have the symbol override so we don't depend on a newer libstdc++, but we need to figure out which libstdc++ version we're using at runtime so we can call the right symbol:

- the shared_ptr overload if we're using libstdc++ 4.9; or
- the shared_ptr, funcptr overload if we're using something different; or
- reimplement more bits using private routines (ick?).

And ideally GCC won't decide to switch back in a future version of libstdc++.

Marking statuses appropriately; we could track this for 62, I guess?

Comment 1

[Nathan Froyd [:froydnj]]

I have a patch for this, just need to convince everything to link libdl appropriately.

Comment 2

[Liz Henry (:lizzard) (relman/hg->git project)]

Tracking to keep an eye on this for 62.

Comment 3

[Mike Hommey [:glandium]]

Should we track this for 61? We may only not catch this on automation because of the libstdc++ version.

Comment 4

[Mike Hommey [:glandium]]

A simpler fix is to stop exporting stl symbols from libxul.

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

Review commit: https://reviewboard.mozilla.org/r/248328/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/248328/

Comment 6

[Nathan Froyd [:froydnj]]

Comment on attachment 8982390 [details]
Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

https://reviewboard.mozilla.org/r/248328/#review254570

This ought to work.  Have you tested with padenot's test re-enabled on try?

Comment 7

[Mike Hommey [:glandium]]

Comment on attachment 8982390 [details]
Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/248328/diff/1-2/

Comment 8

[Mike Hommey [:glandium]]

Comment on attachment 8982390 [details]
Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/248328/diff/2-3/

Comment 9

[Nathan Froyd [:froydnj]]

Comment on attachment 8982390 [details]
Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

https://reviewboard.mozilla.org/r/248328/#review254816

::: python/mozbuild/mozbuild/action/generate_symbols_file.py:75
(Diff revision 3)
>          # A linker version script is generated for GNU LD that looks like the
>          # following:
>          # {

Update this comment to reflect that we're sticking the library name in here now?

Comment 10

[Mike Hommey [:glandium]]

Comment on attachment 8982390 [details]
Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/248328/diff/3-4/

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/838c0ab9cbdb
Don't export libstdc++ symbols with --enable-stdcxx-compat. r=froydnj

Comment 12

[Natalia Csoregi [:nataliaCs]]

Backed out for Linux static bustage.

backout: https://hg.mozilla.org/integration/autoland/rev/74dffd02a249f90777df32ca01579e6c5d3e6a3f

push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=838c0ab9cbdbe85b0f7997ec48289d0a7fc12360&selectedJob=181443402

failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=181443400&repo=autoland&lineNumber=3904

[task 2018-06-01T22:43:34.378Z] 22:43:34     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang++ -std=gnu++14 -o TestNoRefcountedInsideLambdas.o -c -DNDEBUG=1 -DTRIMMED=1 -I/builds/worker/workspace/build/src/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -fsyntax-only -Xclang -verify -ferror-limit=0 -Wno-invalid-noreturn -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -fno-omit-frame-pointer -Werror  -MD -MP -MF .deps/TestNoRefcountedInsideLambdas.o.pp   /builds/worker/workspace/build/src/build/clang-plugin/tests/TestNoRefcountedInsideLambdas.cpp
[task 2018-06-01T22:43:34.378Z] 22:43:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.486Z] 22:43:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.486Z] 22:43:34     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang++ -std=gnu++14 -o TestNonMemMovable.o -c -DNDEBUG=1 -DTRIMMED=1 -I/builds/worker/workspace/build/src/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -fsyntax-only -Xclang -verify -ferror-limit=0 -Wno-invalid-noreturn -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -fno-omit-frame-pointer -Werror  -MD -MP -MF .deps/TestNonMemMovable.o.pp   /builds/worker/workspace/build/src/build/clang-plugin/tests/TestNonMemMovable.cpp
[task 2018-06-01T22:43:34.486Z] 22:43:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T23:23:34.502Z] 23:23:34     INFO - Automation Error: mozprocess timed out after 2400 seconds running ['/usr/bin/python2.7', 'mach', '--log-no-times', 'build', '-v']
[task 2018-06-01T23:23:34.514Z] 23:23:34    ERROR - timed out after 2400 seconds of no output
[task 2018-06-01T23:23:34.514Z] 23:23:34    ERROR - Return code: -15
[task 2018-06-01T23:23:34.514Z] 23:23:34  WARNING - setting return code to 2
[task 2018-06-01T23:23:34.514Z] 23:23:34    FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2018-06-01T23:23:34.514Z] 23:23:34    FATAL - Running post_fatal callback...
[task 2018-06-01T23:23:34.515Z] 23:23:34    FATAL - Exiting -1
[task 2018-06-01T23:23:34.515Z] 23:23:34     INFO - [mozharness: 2018-06-01 23:23:34.515167Z] Finished build step (failed)
[task 2018-06-01T23:23:34.515Z] 23:23:34     INFO - Running post-run listener: _summarize
[task 2018-06-01T23:23:34.515Z] 23:23:34    ERROR - # TBPL FAILURE #
[task 2018-06-01T23:23:34.515Z] 23:23:34     INFO - [mozharness: 2018-06-01 23:23:34.515398Z] FxDesktopBuild summary:
[task 2018-06-01T23:23:34.515Z] 23:23:34    ERROR - # TBPL FAILURE #

Comment 13

[Mike Hommey [:glandium]]

Huh, looks like the clang-plugin didn't like this


*** glibc detected *** /builds/worker/workspace/build/src/clang/bin/clang-5.0: free(): invalid pointer: 0x0000000003c8bf80 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x75bb6)[0x7fb8f696fbb6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7fb8f697495c]
/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so(_ZN26DanglingOnTemporaryChecker5checkERKN5clang12ast_matchers11MatchFinder11MatchResultE+0x8c1)[0x7fb8f6608fa1]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a1bdf]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_ZN5clang12ast_matchers8internal21BoundNodesTreeBuilder12visitMatchesEPNS2_7VisitorE+0x172)[0x29b76d2]
[task 2018-06-01T22:43:34.054Z] 22:43:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.054Z] 22:43:34     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang -std=gnu99 -o TestNANTestingExprC.o -c -DNDEBUG=1 -DTRIMMED=1 -I/builds/worker/workspace/build/src/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -DMOZILLA_CLIENT -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -ffunction-sections -fdata-sections -fno-math-errno -pthread -pipe -fsyntax-only -Xclang -verify -ferror-limit=0 -std=c11 -Wno-invalid-noreturn -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -fno-omit-frame-pointer -Werror  -MD -MP -MF .deps/TestNANTestingExprC.o.pp   /builds/worker/workspace/build/src/build/clang-plugin/tests/TestNANTestingExprC.c
[task 2018-06-01T22:43:34.054Z] 22:43:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.055Z] 22:43:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a1a8b]
[task 2018-06-01T22:43:34.055Z] 22:43:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.055Z] 22:43:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.055Z] 22:43:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29adc2a]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a7cb7]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a4d05]
[task 2018-06-01T22:43:34.056Z] 22:43:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29b0b46]
[task 2018-06-01T22:43:34.058Z] 22:43:34     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang++ -std=gnu++14 -o TestExplicitOperatorBool.o -c -DNDEBUG=1 -DTRIMMED=1 -I/builds/worker/workspace/build/src/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -fsyntax-only -Xclang -verify -ferror-limit=0 -Wno-invalid-noreturn -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -fno-omit-frame-pointer -Werror  -MD -MP -MF .deps/TestExplicitOperatorBool.o.pp   /builds/worker/workspace/build/src/build/clang-plugin/tests/TestExplicitOperatorBool.cpp
[task 2018-06-01T22:43:34.058Z] 22:43:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.058Z] 22:43:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a7cb7]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29aa62c]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a7cb7]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29b4f4a]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a1f11]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[task 2018-06-01T22:43:34.062Z] 22:43:34     INFO -  /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang++ -std=gnu++14 -o TestBadImplicitConversionCtor.o -c -DNDEBUG=1 -DTRIMMED=1 -I/builds/worker/workspace/build/src/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests -I/builds/worker/workspace/build/src/obj-firefox/dist/include -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nspr -I/builds/worker/workspace/build/src/obj-firefox/dist/include/nss -fPIC -DMOZILLA_CLIENT -include /builds/worker/workspace/build/src/obj-firefox/mozilla-config.h -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Qunused-arguments -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -fsyntax-only -Xclang -verify -ferror-limit=0 -Wno-invalid-noreturn -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -fno-omit-frame-pointer -Werror  -MD -MP -MF .deps/TestBadImplicitConversionCtor.o.pp   /builds/worker/workspace/build/src/build/clang-plugin/tests/TestBadImplicitConversionCtor.cpp
[task 2018-06-01T22:43:34.062Z] 22:43:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.062Z] 22:43:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.062Z] 22:43:34     INFO -  make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
[task 2018-06-01T22:43:34.062Z] 22:43:34     INFO -  make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/tests'
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_ZN5clang12ast_matchers11MatchFinder8matchASTERNS_10ASTContextE+0x2cd)[0x29865ad]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_ZN5clang17MultiplexConsumer21HandleTranslationUnitERNS_10ASTContextE+0x2c)[0x1976e9c]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_ZN5clang8ParseASTERNS_4SemaEbb+0x1c6)[0x2057036]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_ZN5clang14FrontendAction7ExecuteEv+0x3f)[0x195215f]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_ZN5clang16CompilerInstance13ExecuteActionERNS_14FrontendActionE+0x418)[0x1901ab8]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_ZN5clang25ExecuteCompilerInvocationEPNS_16CompilerInstanceE+0x1086)[0x19cf0b6]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_Z8cc1_mainN4llvm8ArrayRefIPKcEES2_Pv+0x3ff)[0x7fff6f]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(main+0x2b69)[0x7fe039]
*** glibc detected *** /builds/worker/workspace/build/src/clang/bin/clang-5.0: free(): invalid pointer: 0x0000000003c8bf80 ***
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xfd)[0x7fb8f6918ead]
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x75bb6)[0x7f77e6ddabb6]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f77e6ddf95c]
/builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so(_ZN22KungFuDeathGripChecker5checkERKN5clang12ast_matchers11MatchFinder11MatchResultE+0x643)[0x7f77e6a78c03]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x7fb1f1]
======= Memory map: ========
00400000-03773000 r-xp 00000000 fc:00 1593850                            /builds/worker/workspace/build/src/clang/bin/clang-5.0
03972000-03bea000 rw-p 03372000 fc:00 1593850                            /builds/worker/workspace/build/src/clang/bin/clang-5.0
03bea000-03c9e000 rw-p 00000000 00:00 0 
05971000-05a5b000 rw-p 00000000 00:00 0                                  [heap]
7fb8f0000000-7fb8f0021000 rw-p 00000000 00:00 0 
7fb8f0021000-7fb8f4000000 ---p 00000000 00:00 0 
7fb8f65c0000-7fb8f66de000 r-xp 00000000 fc:00 4234465                    /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so
7fb8f66de000-7fb8f68de000 ---p 0011e000 fc:00 4234465                    /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so
7fb8f68de000-7fb8f68e7000 rw-p 0011e000 fc:00 4234465                    /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so
7fb8f68e7000-7fb8f68fa000 rw-p 00000000 00:00 0 
7fb8f68fa000-7fb8f6a7e000 r-xp 00000000 fc:00 2365198                    /lib/x86_64-linux-gnu/libc-2.13.so
7fb8f6a7e000-7fb8f6c7d000 ---p 00184000 fc:00 2365198                    /lib/x86_64-linux-gnu/libc-2.13.so
7fb8f6c7d000-7fb8f6c81000 r--p 00183000 fc:00 2365198                    /lib/x86_64-linux-gnu/libc-2.13.so
7fb8f6c81000-7fb8f6c82000 rw-p 00187000 fc:00 2365198                    /lib/x86_64-linux-gnu/libc-2.13.so
7fb8f6c82000-7fb8f6c87000 rw-p 00000000 00:00 0 
7fb8f6c87000-7fb8f6c9d000 r-xp 00000000 fc:00 1722333                    /builds/worker/workspace/build/src/clang/lib/libgcc_s.so.1
7fb8f6c9d000-7fb8f6e9c000 ---p 00016000 fc:00 1722333                    /builds/worker/workspace/build/src/clang/lib/libgcc_s.so.1
7fb8f6e9c000-7fb8f6e9d000 rw-p 00015000 fc:00 1722333                    /builds/worker/workspace/build/src/clang/lib/libgcc_s.so.1
7fb8f6e9d000-7fb8f6f1e000 r-xp 00000000 fc:00 2365217                    /lib/x86_64-linux-gnu/libm-2.13.so
7fb8f6f1e000-7fb8f711d000 ---p 00081000 fc:00 2365217                    /lib/x86_64-linux-gnu/libm-2.13.so
7fb8f711d000-7fb8f711e000 r--p 00080000 fc:00 2365217                    /lib/x86_64-linux-gnu/libm-2.13.so
7fb8f711e000-7fb8f711f000 rw-p 00081000 fc:00 2365217                    /lib/x86_64-linux-gnu/libm-2.13.so
7fb8f711f000-7fb8f7135000 r-xp 00000000 fc:00 2365270                    /lib/x86_64-linux-gnu/libz.so.1.2.7
7fb8f7135000-7fb8f7334000 ---p 00016000 fc:00 2365270                    /lib/x86_64-linux-gnu/libz.so.1.2.7
7fb8f7334000-7fb8f7335000 r--p 00015000 fc:00 2365270                    /lib/x86_64-linux-gnu/libz.so.1.2.7
7fb8f7335000-7fb8f7336000 rw-p 00016000 fc:00 2365270                    /lib/x86_64-linux-gnu/libz.so.1.2.7
7fb8f7336000-7fb8f7338000 r-xp 00000000 fc:00 2365206                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fb8f7338000-7fb8f7538000 ---p 00002000 fc:00 2365206                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fb8f7538000-7fb8f7539000 r--p 00002000 fc:00 2365206                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fb8f7539000-7fb8f753a000 rw-p 00003000 fc:00 2365206                    /lib/x86_64-linux-gnu/libdl-2.13.so
7fb8f753a000-7fb8f7541000 r-xp 00000000 fc:00 2365251                    /lib/x86_64-linux-gnu/librt-2.13.so
7fb8f7541000-7fb8f7740000 ---p 00007000 fc:00 2365251                    /lib/x86_64-linux-gnu/librt-2.13.so
7fb8f7740000-7fb8f7741000 r--p 00006000 fc:00 2365251                    /lib/x86_64-linux-gnu/librt-2.13.so
7fb8f7741000-7fb8f7742000 rw-p 00007000 fc:00 2365251                    /lib/x86_64-linux-gnu/librt-2.13.so
7fb8f7742000-7fb8f7759000 r-xp 00000000 fc:00 2365245                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7fb8f7759000-7fb8f7958000 ---p 00017000 fc:00 2365245                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7fb8f7958000-7fb8f7959000 r--p 00016000 fc:00 2365245                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7fb8f7959000-7fb8f795a000 rw-p 00017000 fc:00 2365245                    /lib/x86_64-linux-gnu/libpthread-2.13.so
7fb8f795a000-7fb8f795e000 rw-p 00000000 00:00 0 
7fb8f795e000-7fb8f797e000 r-xp 00000000 fc:00 2365182                    /lib/x86_64-linux-gnu/ld-2.13.so
7fb8f7b39000-7fb8f7b6f000 rw-p 00000000 00:00 0 
7fb8f7b76000-7fb8f7b78000 rw-p 00000000 00:00 0 
7fb8f7b78000-7fb8f7b7b000 r--p 00000000 00:00 0                          [vvar]
7fb8f7b7b000-7fb8f7b7d000 r-xp 00000000 00:00 0                          [vdso]
7fb8f7b7d000-7fb8f7b7e000 r--p 0001f000 fc:00 2365182                    /lib/x86_64-linux-gnu/ld-2.13.so
7fb8f7b7e000-7fb8f7b7f000 rw-p 00020000 fc:00 2365182                    /lib/x86_64-linux-gnu/ld-2.13.so
7fb8f7b7f000-7fb8f7b80000 rw-p 00000000 00:00 0 
7ffdd6bd8000-7ffdd6bfa000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a1bdf]
/builds/worker/workspace/build/src/clang/bin/clang-5.0(_ZN5clang12ast_matchers8internal21BoundNodesTreeBuilder12visitMatchesEPNS2_7VisitorE+0x172)[0x29b76d2]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29a1a8b]
/builds/worker/workspace/build/src/clang/bin/clang-5.0[0x29b0b3b]

Comment 14

[Mike Hommey [:glandium]]

Comment on attachment 8982390 [details]
Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/248328/diff/4-5/

Comment 15

[Mike Hommey [:glandium]]

I'll just go with hiding the _M_start_thread symbols only. Let's figure out the more general std:: thing separately, if ever.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7c74241c18ea
Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat. r=froydnj

Comment 17

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/7c74241c18ea

Comment 18

[Ryan VanderMeulen [:RyanVM]]

Hey Mike, what branches need uplift here?

Comment 19

[Mike Hommey [:glandium]]

Comment on attachment 8982390 [details]
Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

Approval Request Comment
[Feature/Bug causing the regression]: bug 1389435 (switching the build to GCC 6), which means this has been a problem since 57.
[User impact if declined]: possible infinite loop when creating /some/ kinds of threads, although it doesn't seem to be happening in the wild (according to crash-stats).
[Is this code covered by automated tests?]: The test for bug 1454385 covers it (this is how this was found in the first place).
[Has the fix been verified in Nightly?]: Yes, by reenabling the test for bug 1454385
[Needs manual test from QE? If yes, steps to reproduce]: N/A
[List of other uplifts needed for the feature/fix]: N/A
[Is the change risky?]: Not much
[Why is the change risky/not risky?]: There is a change in how symbols are exported from libraries such as libmozavcodec.so, but that would tend to make things stricter than what they were before, forcing symbols being resolved from those libraries rather than system libraries that might have the same symbols, so overall, this should be a win.
[String changes made/needed]: N/A

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 8982390 [details]
Bug 1464084 - Don't export std::thread::_M_start_thread symbols with --enable-stdcxx-compat.

Fixes a possible infinite loop caused by the switch to gcc6. Verified by the test on trunk which was previously failing due to this bug and is now successfully re-enabled in production. Approved for 61.0b12 and ESR 60.1.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/4f9a0ee0ac42

Comment 22

[Ryan VanderMeulen [:RyanVM]]

Backed out for SM(pkg) libc-2.13.so crashes (seen on top of both Beta and ESR60).
https://treeherder.mozilla.org/logviewer.html#?job_id=182006189&repo=mozilla-beta

https://hg.mozilla.org/releases/mozilla-beta/rev/2e763d5ed8e5eba9cba54780a3142b9365b0cce3

Comment 23

[Mike Hommey [:glandium]]

The more I look at it, the more I'm puzzled.

On the failing build, this is what happens per LD_DEBUG=all:
     24392:     symbol=malloc;  lookup in file=./dist/bin/jsapi-tests [0]
     24392:     symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libpthread.so.0 [0]
     24392:     symbol=malloc;  lookup in file=/builds/worker/workspace/sm-package/mozjs-60.0.2/obj-spider/dist/bin/libnspr4.so [0]
     24392:     symbol=malloc;  lookup in file=/builds/worker/workspace/sm-package/mozjs-60.0.2/obj-spider/dist/bin/libplc4.so [0]
     24392:     symbol=malloc;  lookup in file=/builds/worker/workspace/sm-package/mozjs-60.0.2/obj-spider/dist/bin/libplds4.so [0]
     24392:     symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libdl.so.2 [0]
     24392:     symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/librt.so.1 [0]
     24392:     symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libz.so.1 [0]
     24392:     symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libm.so.6 [0]
     24392:     symbol=malloc;  lookup in file=/usr/lib/x86_64-linux-gnu/libstdc++.so.6 [0]
     24392:     symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libgcc_s.so.1 [0]
     24392:     symbol=malloc;  lookup in file=/lib/x86_64-linux-gnu/libc.so.6 [0]
     24392:     binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to /lib/x86_64-linux-gnu/libc.so.6 [0]: normal symbol `malloc' [GLIBC_2.2.5]

That is, mozjemalloc is not used.

On a non failing build (with the backout):
      2276:     symbol=malloc;  lookup in file=./dist/bin/jsapi-tests [0]
      2276:     binding file /lib/x86_64-linux-gnu/libc.so.6 [0] to ./dist/bin/jsapi-tests [0]: normal symbol `malloc' [GLIBC_2.2.5]

so malloc is taken from jsapi-tests as expected, and is thus mozjemalloc.

But here is the crazy thing: in both cases, malloc is exported from jsapi-tests:
broken:
  $ objdump -T dist/bin/jsapi-tests | grep \ malloc$
  000000000051e400 g    DF .text  0000000000000350  Base        malloc
non-broken:
  $ objdump -T dist/bin/jsapi-tests | grep \ malloc$
  000000000051e5a0 g    DF .text  0000000000000350  Base        malloc

The LD_DEBUG=all output above are the first lines that differ between both cases. There is no difference between the logs before that.

And if I download jsapi-tests from a mozilla-central sm-pkg build, malloc is correctly bound to jsapi-tests!

Comment 24

[Mike Hommey [:glandium]]

So whatever gold is doing to break this, it's because of gold being used for those builds. Which means there are two things to figure out:
- why the sm-pkg builds on beta are using gold but the ones on central aren't?
- what is the hidden difference between what gold and bfd ld produce that makes ld.so not find malloc in jsapi-tests?

Comment 25

[Mike Hommey [:glandium]]

> - why the sm-pkg builds on beta are using gold but the ones on central aren't?

That's because bug 1455767 apparently broke defaulting to gold on local builds. Which brings to the problem that the sm-pkg builds shouldn't be using gold anyways.

Comment 26

[Mike Hommey [:glandium]]

So the problem is that gold doesn't do what would seem to be the right thing when it creates symbol versions (at least, it does things differently than both bfd ld and lld), *and* both objdump and readelf mask the problem by not displaying versions correctly in their output of the symbol table: they show the same thing as if there was no symbol version even when there is.

Comment 27

[Mike Hommey [:glandium]]

Filed:
https://bugs.llvm.org/show_bug.cgi?id=37724
https://sourceware.org/bugzilla/show_bug.cgi?id=23267
https://sourceware.org/bugzilla/show_bug.cgi?id=23268

Comment 28

[Mike Hommey [:glandium]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #22)
> Backed out for SM(pkg) libc-2.13.so crashes (seen on top of both Beta and
> ESR60).
> https://treeherder.mozilla.org/logviewer.html#?job_id=182006189&repo=mozilla-
> beta
> 
> https://hg.mozilla.org/releases/mozilla-beta/rev/
> 2e763d5ed8e5eba9cba54780a3142b9365b0cce3

These go away thanks to bug 1467041. Can this reland?

Comment 29

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/4c893cb98224

Comment 30

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/123bd919d459