GLIBCXX_3.4.9 not found during 'make check' on linux builder

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
7 years ago
3 years ago

People

(Reporter: vlad, Assigned: rail)

Tracking

({relnote})

other
All
Linux
relnote
Dependency tree / graph

Firefox Tracking Flags

(blocking2.0 betaN+)

Details

(Whiteboard: [linux64])

Attachments

(2 attachments, 4 obsolete attachments)

Similar to bug 560894, but somewhat different; nthomas asked for a new bug.

My patch introduces a new symbol dependency from libcxx that has an explicit 3.4.9 version.  'make check' fails for precisely two tests, both of them xpcom:

make[2]: Entering directory `/builds/slave/tryserver-linux64/build/obj-firefox/xpcom/tests'
../../dist/bin/ShowAlignments: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by ../../dist/bin/libxul.so)
make[2]: *** [check] Error 1
Running XPIDL tests
/builds/slave/tryserver-linux64/build/obj-firefox/dist/bin/xpidl -m header /builds/slave/tryserver-linux64/build/xpcom/tests/TestScriptable.idl
rm -f ../../dist/bin/components/compreg.dat; \
	 \
	XPCOM_DEBUG_BREAK=stack-and-abort ../../dist/bin/run-mozilla.sh \
	  ../../dist/bin/TestRegistrationOrder "/builds/slave/tryserver-linux64/build/xpcom/tests/regorder";
../../dist/bin/TestRegistrationOrder: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by ../../dist/bin/libxul.so)
make[2]: *** [check] Error 1
make[2]: Leaving directory `/builds/slave/tryserver-linux64/build/obj-firefox/xpcom/tests'

(From http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1279142586.1279148302.28883.gz)

Note that it's getting "/usr/lib64/libstdc++.so.6".

I don't see LD_LIBRARY_PATH being set anywhere, though nthomas says that he tried doing a manual make check with LD_LIBRARY_PATH set:

18:27 < nthomas> |LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib make -k check| also fails on that slave
'LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib64 make -k check' does work though, and the scope is limited make check (on try anyway).
Created attachment 457586 [details] [diff] [review]
set LD_LIBRARY_PATH in linux64 env for try

so is this all that's needed? or do we want this for all linux64?
Assignee: nobody → lsblakk
Attachment #457586 - Flags: review?(nrthomas)
this is happening for me on the n810s

INFO | automation.py | Application pid: 1602
/builds/fennec/fennec: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /builds/fennec/libxul.so)
TEST-UNEXPECTED-FAIL | automation.py | Exited with code 1 during test run
INFO | automation.py | Application ran for: 0:00:00.195434
INFO | automation.py | Reading PID log: /var/tmp/tmpeC37hipidlog
WARNING | automationutils.processLeakLog() | refcount logging is off, so leaks can't be detected!

REFTEST INFO | runreftest.py | Running tests: end.
Splitting mobile concerns into bug 579186
Comment on attachment 457586 [details] [diff] [review]
set LD_LIBRARY_PATH in linux64 env for try

ANGLE landed on m-c so we'll need that as well, and should probably do this on any branch that's using gcc 4.3.3. And I'd really love if we could do this in the mozconfig, so that changes like bug 559964 can be done in one place.
Attachment #457586 - Flags: review?(nrthomas) → review-
blocking2.0: --- → betaN+
unassigning myself since I will be on vacation next week and hopefully someone can get on this sooner than I can.
Assignee: lsblakk → nobody

Updated

7 years ago
Whiteboard: [linux64]
Any progress on this?

Comment 8

7 years ago
grabbing this bug to check it's status and even possible land it
Assignee: nobody → bear

Comment 9

7 years ago
Created attachment 464104 [details] [diff] [review]
set LD_LIBRARY_PATH in linux64 mozconfig
Attachment #464104 - Flags: review?(lsblakk)
Attachment #464104 - Flags: feedback?(nrthomas)

Comment 10

7 years ago
(In reply to comment #5)
> Comment on attachment 457586 [details] [diff] [review]
> set LD_LIBRARY_PATH in linux64 env for try
> 
> ANGLE landed on m-c so we'll need that as well, and should probably do this on
> any branch that's using gcc 4.3.3. And I'd really love if we could do this in
> the mozconfig, so that changes like bug 559964 can be done in one place.

Nick,
I mentioned to Ted about making the change attached above and he reminded me that LD_LIBRARY_PATH is not "captured" by configure and persisted thru the to the makefiles.
I also see that in the 32bit version of this change we are setting the value in config.py, both of these are leading me to want to make the change in config.py
Status: NEW → ASSIGNED
Comment on attachment 464104 [details] [diff] [review]
set LD_LIBRARY_PATH in linux64 mozconfig

Bummer, we'll have to use config.py then. We'll need to give Rail a headsup in bug 559964.
Attachment #464104 - Attachment is obsolete: true
Attachment #464104 - Flags: review?(lsblakk)
Attachment #464104 - Flags: feedback?(nrthomas)
(Assignee)

Comment 12

7 years ago
(In reply to comment #11)
> Bummer, we'll have to use config.py then. We'll need to give Rail a headsup in
> bug 559964.

Hopefully it won't affect the gcc45, which was compiled with -rpath set to its own directory: http://hg.mozilla.org/build/rpm-sources/file/c8fe426b3bdb/gcc45/centos5-i686/gcc45.spec#l51

Comment 13

7 years ago
Created attachment 464401 [details] [diff] [review]
[WIP] set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug

Does not yet include other branches than try - my running of builder_list.py on sm02 is failing and I have to work out why
Attachment #457586 - Attachment is obsolete: true
Attachment #464401 - Flags: feedback?(nrthomas)
Comment on attachment 464401 [details] [diff] [review]
[WIP] set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug

>diff --git a/mozilla/config.py b/mozilla/config.py
>+BRANCHES['tryserver']['platforms']['linux']['env']['LD_LIBRARY_PATH'] = '/tools/gcc-4.3.3/installed/lib'
>+BRANCHES['tryserver']['platforms']['linux-debug']['env']['LD_LIBRARY_PATH'] ='/tools/gcc-4.3.3/installed/lib:%s/dist/bin' % OBJDIR

Is this for parity with 64bit ?

>+BRANCHES['tryserver']['platforms']['linux64']['env']['LD_LIBRARY_PATH'] = '/tools/gcc-4.3.3/installed/lib'
>+BRANCHES['tryserver']['platforms']['linux64-debug']['env']['LD_LIBRARY_PATH'] ='/tools/gcc-4.3.3/installed/lib:%s/dist/bin' % OBJDIR

Should be lib64 I think.

>+BRANCHES['tryserver']['platforms']['linux']['unittest-env'] = {
>+    'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',
>+}
>+BRANCHES['tryserver']['platforms']['linux64']['unittest-env'] = {
>+    'LD_LIBRARY_PATH': '/tools/gcc-4.3.3/installed/lib',
>+}

Where does this get used ? If it's old-style unit test builders then those are dead as a dodo for try, and only survive on 1.9.1 and 1.9.2.

The other thing is that we'd want to make these changes just about everywhere except 1.9.1 and 1.9.2. Everything else merges back and forth with mozilla-central.
Attachment #464401 - Flags: feedback?(nrthomas) → feedback-
beg'pardon on the other branches, missed your comment on that.
Any progress on this?

Updated

7 years ago
Priority: -- → P2
Looks like this is happening on x86 as well

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1284148927.1284150569.14341.gz#err6
Blocks: 590181
Hardware: x86_64 → All
Summary: GLIBCXX_3.4.9 not found during 'make check' on linux x86-64 builder → GLIBCXX_3.4.9 not found during 'make check' on linux builder
My push from 590181 seems to trigger this consistently.  Any chance of getting fixed soon?
Please keep in mind that using LD_LIBRARY_PATH will only work around the root problem, which is that the resulting executable won't run on systems with an old enough libstdc++, which is exactly what happens with these tests: the systems the test run on apparently use an old enough libstdc++.

Now the question is, apart from the build and testing environments, are there users who would be impacted ? More notably, when mozilla switches to gcc 4.5+, there are chances this will get even worse from that perspective, and while libstdc++ from gcc 4.3 is probably available to masses (but maybe not on exotic platforms (android? maemo?)), libstdc++ from gcc 4.5+ is probably not widespread enough.
Any progress on this?

Comment 21

7 years ago
(In reply to comment #20)
> Any progress on this?

Does 4.3 have -static-libstdc++? if it doesn't then it's a matter of modifying our mozconfig + recompiling gcc with CXXFLAGS=-fPIC just like we do for 4.5 in bug 559964. If that flag isn't available then we should block this on deploying gcc 4.5.
(In reply to comment #20)
> Any progress on this?

vlad - any response to comment 19?  Unless we switch to statically linking libstdc++, users with older stdc++ libraries will be broken.

Comment 23

7 years ago
(In reply to comment #22)
> (In reply to comment #20)
> > Any progress on this?
> 
> vlad - any response to comment 19?  Unless we switch to statically linking
> libstdc++, users with older stdc++ libraries will be broken.

I'm not Vlad, but this is only fixed by statically linking stdc++. I don't think there is another way. Unfortunately this requires rebuilding gcc 4.3 with CXXFLAGS=-fPIC(which we do for gcc 4.5 builds already). So I'm hoping we can just switch to gcc 4.5 in bug 559964 real soon and let 4.3 RIP.
What Taras said; I guess we're waiting on bug 559964 which should fix this?
This is preventing me from landing bug 594489 which is a serious WebGL bug (makes many WebGL demos fail on non-US locales).
Blocks: 594489
This is also preventing me from landing bug 599969.
Depends on: 599969

Updated

7 years ago
Blocks: 599969
No longer depends on: 599969
This will shortly be preventing the landing of bug 603333, which is a security bug.  It sounds like this/bug 559964 is waiting for b7 to ship..
Blocks: 603333

Updated

7 years ago
Assignee: bear → nobody
Going to find an owner for this in the releng meeting today.
Taras told me that bug 559964 should fix this. Rail, can you confirm that/check?
Assignee: nobody → rail
The places branch is now perma-orange (linux opt, make check) because of this and the work that caused this to show up is needed.  Is there a revised ETA for this to get resolved?
Also, I'm not seeing this on just linux64, but also the linux32 machines.
Blocked on switching over to gcc4.5 I believe.
Please then fill the 'Depends on' field :-)
I'm assuming that's bug 559964, which was until now marked as b2.0-? :(
Depends on: 559964
So, bug 559964 isn't sticking.  Is there anything else we can do about this so I can get the places branch green again (apart from code changes, preferably)
There's a decent chance some relatively simple code changes could avoid it.  That said it's somewhat hard to tell what they would be given that the error on the places branch is in a test that never leaves the build machine (i.e., not packaged).
(In reply to comment #36)
> There's a decent chance some relatively simple code changes could avoid it. 
> That said it's somewhat hard to tell what they would be given that the error on
> the places branch is in a test that never leaves the build machine (i.e., not
> packaged).
Well, I'm quite certain that it's because of the do_check_eq function using string streams:
http://hg.mozilla.org/projects/places/file/ecdd25bb9d90/storage/test/storage_test_harness.h#l82

It's just hard to have useful failure output without using it (AFAIK).
(In reply to comment #36)
> That said it's somewhat hard to tell what they would be given that the error on
> the places branch is in a test that never leaves the build machine (i.e., not
> packaged).

One could send a hack to try that would add the test to the package manifest.
(In reply to comment #38)
> One could send a hack to try that would add the test to the package manifest.
But how would packaging it help?
(In reply to comment #39)
> But how would packaging it help?

Packaging it gives you the binary. With the binary, you can know what symbol exactly needs version GLIBCXX_3.4.9.

Comment 41

7 years ago
(In reply to comment #40)
> (In reply to comment #39)
> > But how would packaging it help?
> 
> Packaging it gives you the binary. With the binary, you can know what symbol
> exactly needs version GLIBCXX_3.4.9.

Mike,
we need static-libstdc++ for other bugs too, so it doesn't matter.
Rail, what's your feeling on an ETA for this? It's blocking a bunch of performance work on the Places branch.
(In reply to comment #42)
> Rail, what's your feeling on an ETA for this? It's blocking a bunch of
> performance work on the Places branch.

Do you really want to go ahead with the code knowing that it breaks compatibility with older libstdc++ versions?  Until gcc-4.5 with static linking finally sticks, aren't these errors useful indications of code that's going to break on users' machines?
(In reply to comment #43)
> Do you really want to go ahead with the code knowing that it breaks
> compatibility with older libstdc++ versions?  Until gcc-4.5 with static linking
> finally sticks, aren't these errors useful indications of code that's going to
> break on users' machines?
Not in the Places branch case at least.  It's test-only code, but there isn't any way to do what I'm doing otherwise without writing lots of code for each type.

Comment 45

7 years ago
(In reply to comment #44)
> (In reply to comment #43)
> > Do you really want to go ahead with the code knowing that it breaks
> > compatibility with older libstdc++ versions?  Until gcc-4.5 with static linking
> > finally sticks, aren't these errors useful indications of code that's going to
> > break on users' machines?
> Not in the Places branch case at least.  It's test-only code, but there isn't
> any way to do what I'm doing otherwise without writing lots of code for each
> type.

On the branch you can just copy the appropriate libstdc++.so.whatever into your dist/firefox
(Assignee)

Comment 46

7 years ago
> Rail, what's your feeling on an ETA for this? It's blocking a bunch of
> performance work on the Places branch.

We tried to switch to gcc-4.5 last week for mozilla-central, but had to backout.

> On the branch you can just copy the appropriate libstdc++.so.whatever into your
> dist/firefox

...or we can use gcc-4.5 as a default compiler for places branch. What do you think?
(In reply to comment #46)
> ...or we can use gcc-4.5 as a default compiler for places branch. What do you
> think?
We'll still turn mozilla-central orange once we merge back into it.

Comment 48

7 years ago
Shawn, by that time we should have 4.5 deployed. I suspect even if the freeze isn't fixed we can figure out which files are causing the hang to avoid it..ie there is no hang without using -O3
(In reply to comment #48)
> Shawn, by that time we should have 4.5 deployed. I suspect even if the freeze
> isn't fixed we can figure out which files are causing the hang to avoid it..ie
> there is no hang without using -O3
So end of this week is a reasonable timeline?
(Assignee)

Comment 50

7 years ago
Created attachment 493200 [details] [diff] [review]
set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug (tryserver and places)

Attached is refreshed bear's patch applied for tryserver and places branches. The same approach is also used for electrolysis branch (and it uses lib directory for linux64, not lib64).
Attachment #464401 - Attachment is obsolete: true
Attachment #493200 - Flags: review?(bhearsum)
Comment on attachment 493200 [details] [diff] [review]
set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug (tryserver and places)

Why does linux64-debug use lib64 and linux64 use lib? Don't we need this for mozilla-central and all of the other branches based on it, too? It's particularly concerning that this patch adjusts tryserver and not mozilla-central.

Do we need to fix the electrolysis entries, too? It's missing LD_LIBRARY_PATH changes for ['linux64']['env'].
Blocks: 614288
(Assignee)

Comment 52

7 years ago
Created attachment 493399 [details] [diff] [review]
set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug (all gcc-4.3 branches)

(In reply to comment #51)
 
> Why does linux64-debug use lib64 and linux64 use lib? Don't we need this for
> mozilla-central and all of the other branches based on it, too? It's
> particularly concerning that this patch adjusts tryserver and not
> mozilla-central.
> 
> Do we need to fix the electrolysis entries, too? It's missing LD_LIBRARY_PATH
> changes for ['linux64']['env'].

I removed all previous LD_LIBRARY_PATH entries for electrolysis and added a block at the end which sets LD_LIBRARY_PATH for all gcc-4.3 branches (linux, linux64, linux-debug, linux64-debug).

BRANCHES[branch]['platforms']['linux64-debug']['unittest-env'] is not being used anywhere probably, but I set it for consistency.
Attachment #493399 - Flags: review?(catlee)
Comment on attachment 493200 [details] [diff] [review]
set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug (tryserver and places)

Looks like this one is obsolete now
Attachment #493200 - Attachment is obsolete: true
Attachment #493200 - Flags: review?(bhearsum)
Comment on attachment 493399 [details] [diff] [review]
set LD_LIBRARY_PATH in branch platform config environments for linux, linux64 and -debug (all gcc-4.3 branches)

So...technically this looks fine.

I'm *REALLY* nervous about switching this on for mozilla-central since it means that code that depends on newer libstdc++ can sneak in.
Attachment #493399 - Flags: review?(catlee) → review+
it's already sneaking in, because we're using a compiler that requires a newer libstdc++!
(Assignee)

Comment 56

7 years ago
According to IRC, we should update system requirements for Firefox 4.0. Setting relnote keyword.
Flags: needs-reconfig?
Keywords: relnote
Planning to do a reconfig with this tomorrow morning.
Flags: needs-reconfig? → needs-reconfig+
Attachment #493399 - Flags: checked-in+
Masters have been updated.
Flags: needs-reconfig+
(Assignee)

Comment 59

7 years ago
All done here. Feel free to reopen if the same problem appears.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Doesn't look fixed:

make[2]: Entering directory `/builds/slave/tryserver-linux64/build/obj-firefox/xpcom/tests'
../../dist/bin/ShowAlignments: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by ../../dist/bin/libxul.so)
make[2]: *** [check] Error 1

when re-enabling ANGLE.  Only linux-64 opt failed though, linux-32 opt/debug and linux-64 debug both succeeded.  Here's the failing log.

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1292286127.1292293295.5858.gz&fulltext=1

on try-linux64-slave04.build.mozilla.org.  I see LD_LIBRARY_PATH being set to   LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib -- shouldn't that be lib64?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
blocking2.0: betaN+ → beta9+
^ ping?
(Assignee)

Comment 62

7 years ago
pong. Sorry for the delay with the reply, all hands, beta8, travel, beta8... :)

> on try-linux64-slave04.build.mozilla.org.  I see LD_LIBRARY_PATH being set to  
> LD_LIBRARY_PATH=/tools/gcc-4.3.3/installed/lib -- shouldn't that be lib64?

Yeah, my fault, just checked it to be sure:

$ file /tools/gcc-4.3.3/installed/lib/libstdc++.so.6.0.10 
/tools/gcc-4.3.3/installed/lib/libstdc++.so.6.0.10: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), not stripped
$ file /tools/gcc-4.3.3/installed/lib64/libstdc++.so.6.0.10 
/tools/gcc-4.3.3/installed/lib64/libstdc++.so.6.0.10: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), not stripped


Patch incoming.
(Assignee)

Comment 63

7 years ago
Created attachment 499446 [details] [diff] [review]
Use lib64 for linux64
Attachment #499446 - Flags: review?(bhearsum)
Attachment #499446 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 64

7 years ago
Comment on attachment 499446 [details] [diff] [review]
Use lib64 for linux64

http://hg.mozilla.org/build/buildbot-configs/rev/38e0eed9107c
Attachment #499446 - Flags: checked-in+
(Assignee)

Comment 65

7 years ago
http://hg.mozilla.org/build/buildbot-configs/rev/38e0eed9107c should be landed merged to production.
Flags: needs-reconfig?

Updated

7 years ago
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Flags: needs-reconfig?
Resolution: --- → FIXED
Depends on: 621704
Depends on: 579186

Updated

7 years ago
No longer blocks: 603333

Comment 66

7 years ago
As per today's meeting, beta 9 will be a time-based release. Marking these all betaN+. Please move it back to beta9+ if you believe it MUST be in the next beta (ie: trunk is in an unshippable state without this)
No longer blocks: 590181, 578877, 594489, 599969, 614288
blocking2.0: beta9+ → betaN+
No longer depends on: 559964, 579186, 621704
Keywords: relnote

Updated

7 years ago
Depends on: 621704
Depends on: 559964, 579186
Keywords: relnote

Updated

7 years ago
Blocks: 603333
No longer blocks: 590181
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.