610311 - In-tree minidump_stackwalk requires libraries not present on slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jesse Ruderman]

Trying to use http://hg.mozilla.org/build/tools/raw-file/default/breakpad/linux/minidump_stackwalk on one of the test slaves:

/builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk)

/builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.11' not found (required by /builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk)

minidump_stackwalk exited with return code 1

Comment 1

[Bob Clary [:bc] (inactive)]

What version of cygwin is used on the test slaves? The one I built was with cygwin 1.7. It should be pretty easy to rebuild with CXXFLAGS="-g -O1" ./configure if the slaves are still using Cygiwn 1.5.

Comment 2

[(not currently active) Ted Mielczarek]

This isn't Cygwin, this is Linux. We've hit this a number of times in the past, see bug 578880, bug 526868, bug 554854.

Comment 3

[Mike Taylor [:bear]]

Ted, is this an issue for running the minidump_stackwalk manually that environment variables need to be set appropriately or built for the environment he is testing in?

In otherwords I'm having trouble seeing exactly how RelEng would help solve this?

Comment 4

[(not currently active) Ted Mielczarek]

We had this problem running test binaries (probably one of those bugs I linked in comment 2), and we needed to set LD_LIBRARY_PATH in the build environment for it to work, since we're building with a newer libstdc++ thatn what's installed on the slaves.

Comment 5

[John Ford [:jhford] CET/CEST Berlin Time]

Is this on a fedora slave?  I see that the libstdc++.so.6 library on a standard slave does have both of those strings.  Is there something more than the below command output to verify the status of this library?

[cltbld@talos-r3-fed-001 ~]$ strings /usr/lib/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_FORCE_NEW
GLIBCXX_DEBUG_MESSAGE_LENGTH

Comment 6

[Jesse Ruderman]

I'm not sure how to get from a fuzzing log to a slave name.  Should I add something that writes the slave name to a file?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

The fuzzing jobs run on the CentOS builders, which will have an older libstdc++ on them.

Comment 8

[Jesse Ruderman]

It would be nice to get this fixed so I can find Linux-specific crash bugs through fuzzing :)

Comment 9

[Mike Taylor [:bear]]

Assigning this to Rail.

Rail, can you look at the issue here - it seems to me that we can set the LD_LIBRARY_PATH env var to point to the GCC environment this script needs.

Can you confirm that? and if so, then make a patch to change this script to have that information?

thanks!

Comment 10

[Rail Aliiev [:rail]]

IS it possible to use statically linked minidump_stackwalk? In this case we can run it on any slave.

Comment 11

[(not currently active) Ted Mielczarek]

Yeah, there shouldn't be any problem with that, I just didn't build it that way.

Comment 12

[Rail Aliiev [:rail]]

(In reply to comment #11)
> Yeah, there shouldn't be any problem with that, I just didn't build it that
> way.

Assigning to you in this case. ;)

Feel free to push it back to me when we are ready for deployment of new binaries.

Comment 13

[(not currently active) Ted Mielczarek]

Attachment: Statically link Linux minidump_stackwalk

Here's a patch to build/tools with a statically-linked Linux minidump_stackwalk.

Comment 14

[(not currently active) Ted Mielczarek]

Attachment: Statically link Linux minidump_stackwalk

Here's a patch to build/tools with a statically-linked Linux minidump_stackwalk.

Comment 15

[[e-mail] (:bhearsum)]

Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Welcome to stampy town.

Comment 16

[Chris Cooper [:coop] (he/him)]

Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

http://hg.mozilla.org/build/tools/rev/776728aff443

Comment 17

[Jesse Ruderman]

Still broken, at least for 64-bit:

/builds/slave/fuzzer-linux64/build/minidump_stackwalk: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /builds/slave/fuzzer-linux64/build/minidump_stackwalk)

Comment 18

[Jesse Ruderman]

Does seem to be working on 32-bit :)

Comment 19

[(not currently active) Ted Mielczarek]

Ah, crap, I apparently only fixed the 32-bit mdsw. I thought they were symlinks at one point?

Comment 20

[Jesse Ruderman]

Having this working on 32-bit allowed me to find bug 673390 :)

Comment 21

[Chris Cooper [:coop] (he/him)]

(In reply to comment #19)
> Ah, crap, I apparently only fixed the 32-bit mdsw. I thought they were
> symlinks at one point?

Ted: does this imply that I can simply copy over the already-built statically linked version into linux64 and it will just work?

Comment 22

[(not currently active) Ted Mielczarek]

That ought to be fine, if it's statically linked then it shouldn't have any library dependencies to worry about, so it should work.

Comment 23

[Chris Cooper [:coop] (he/him)]

Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Checked-in for linux64 also:

http://hg.mozilla.org/build/tools/rev/9e0852356108

Comment 24

[Mike Hommey [:glandium]]

I don't think this works. If I recall correctly, a linux32 executable is only able to deal with linux32 at runtime. So using the linux32 executable on linux64 is not going to work.

Comment 25

[(not currently active) Ted Mielczarek]

It runs fine on my 64-bit Ubuntu system, FWIW.

Comment 26

[Mike Hommey [:glandium]]

(In reply to comment #25)
> It runs fine on my 64-bit Ubuntu system, FWIW.

I must be mixing with dump_symbols.