In-tree minidump_stackwalk requires libraries not present on slaves

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
7 years ago
4 years ago

People

(Reporter: Jesse Ruderman, Assigned: ted)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [automation][sg:want P4])

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

7 years ago
Trying to use http://hg.mozilla.org/build/tools/raw-file/default/breakpad/linux/minidump_stackwalk on one of the test slaves:

/builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk)

/builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.11' not found (required by /builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk)

minidump_stackwalk exited with return code 1

Comment 1

7 years ago
What version of cygwin is used on the test slaves? The one I built was with cygwin 1.7. It should be pretty easy to rebuild with CXXFLAGS="-g -O1" ./configure if the slaves are still using Cygiwn 1.5.
(Assignee)

Comment 2

7 years ago
This isn't Cygwin, this is Linux. We've hit this a number of times in the past, see bug 578880, bug 526868, bug 554854.

Comment 3

7 years ago
Ted, is this an issue for running the minidump_stackwalk manually that environment variables need to be set appropriately or built for the environment he is testing in?

In otherwords I'm having trouble seeing exactly how RelEng would help solve this?
Whiteboard: [triagefollowup]
(Assignee)

Comment 4

7 years ago
We had this problem running test binaries (probably one of those bugs I linked in comment 2), and we needed to set LD_LIBRARY_PATH in the build environment for it to work, since we're building with a newer libstdc++ thatn what's installed on the slaves.
Is this on a fedora slave?  I see that the libstdc++.so.6 library on a standard slave does have both of those strings.  Is there something more than the below command output to verify the status of this library?

[cltbld@talos-r3-fed-001 ~]$ strings /usr/lib/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_FORCE_NEW
GLIBCXX_DEBUG_MESSAGE_LENGTH
(Reporter)

Comment 6

7 years ago
I'm not sure how to get from a fuzzing log to a slave name.  Should I add something that writes the slave name to a file?
The fuzzing jobs run on the CentOS builders, which will have an older libstdc++ on them.
(Reporter)

Comment 8

7 years ago
It would be nice to get this fixed so I can find Linux-specific crash bugs through fuzzing :)

Comment 9

7 years ago
Assigning this to Rail.

Rail, can you look at the issue here - it seems to me that we can set the LD_LIBRARY_PATH env var to point to the GCC environment this script needs.

Can you confirm that? and if so, then make a patch to change this script to have that information?

thanks!
Assignee: nobody → rail
IS it possible to use statically linked minidump_stackwalk? In this case we can run it on any slave.
(Assignee)

Comment 11

7 years ago
Yeah, there shouldn't be any problem with that, I just didn't build it that way.
(In reply to comment #11)
> Yeah, there shouldn't be any problem with that, I just didn't build it that
> way.

Assigning to you in this case. ;)

Feel free to push it back to me when we are ready for deployment of new binaries.
Assignee: rail → ted.mielczarek

Updated

7 years ago
Whiteboard: [triagefollowup] → [automation]
(Assignee)

Comment 13

7 years ago
Created attachment 497624 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Here's a patch to build/tools with a statically-linked Linux minidump_stackwalk.
(Assignee)

Updated

7 years ago
Attachment #497624 - Attachment is obsolete: true
(Assignee)

Comment 14

7 years ago
Created attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Here's a patch to build/tools with a statically-linked Linux minidump_stackwalk.
(Assignee)

Updated

7 years ago
Attachment #497626 - Flags: review?(bhearsum)
Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Welcome to stampy town.
Attachment #497626 - Flags: review?(bhearsum) → review+
(Reporter)

Updated

6 years ago
Whiteboard: [automation] → [automation][sg:want P4]
Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

http://hg.mozilla.org/build/tools/rev/776728aff443
Attachment #497626 - Flags: checked-in+
(Assignee)

Updated

6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 17

6 years ago
Still broken, at least for 64-bit:

/builds/slave/fuzzer-linux64/build/minidump_stackwalk: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /builds/slave/fuzzer-linux64/build/minidump_stackwalk)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
(Reporter)

Comment 18

6 years ago
Does seem to be working on 32-bit :)
(Assignee)

Comment 19

6 years ago
Ah, crap, I apparently only fixed the 32-bit mdsw. I thought they were symlinks at one point?
(Reporter)

Comment 20

6 years ago
Having this working on 32-bit allowed me to find bug 673390 :)
(In reply to comment #19)
> Ah, crap, I apparently only fixed the 32-bit mdsw. I thought they were
> symlinks at one point?

Ted: does this imply that I can simply copy over the already-built statically linked version into linux64 and it will just work?
(Assignee)

Comment 22

6 years ago
That ought to be fine, if it's statically linked then it shouldn't have any library dependencies to worry about, so it should work.
Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Checked-in for linux64 also:

http://hg.mozilla.org/build/tools/rev/9e0852356108

Updated

6 years ago
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
I don't think this works. If I recall correctly, a linux32 executable is only able to deal with linux32 at runtime. So using the linux32 executable on linux64 is not going to work.
(Assignee)

Comment 25

6 years ago
It runs fine on my 64-bit Ubuntu system, FWIW.
(In reply to comment #25)
> It runs fine on my 64-bit Ubuntu system, FWIW.

I must be mixing with dump_symbols.
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.