Last Comment Bug 610311 - In-tree minidump_stackwalk requires libraries not present on slaves
: In-tree minidump_stackwalk requires libraries not present on slaves
Status: RESOLVED FIXED
[automation][sg:want P4]
:
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: x86 Linux
: -- normal (vote)
: ---
Assigned To: Ted Mielczarek [:ted.mielczarek]
:
:
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2010-11-07 21:28 PST by Jesse Ruderman
Modified: 2013-08-12 21:54 PDT (History)
9 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Statically link Linux minidump_stackwalk (781.75 KB, patch)
2010-12-14 14:45 PST, Ted Mielczarek [:ted.mielczarek]
no flags Details | Diff | Splinter Review
Statically link Linux minidump_stackwalk (781.75 KB, patch)
2010-12-14 14:46 PST, Ted Mielczarek [:ted.mielczarek]
bhearsum: review+
coop: checked‑in+
Details | Diff | Splinter Review

Description User image Jesse Ruderman 2010-11-07 21:28:46 PST
Trying to use http://hg.mozilla.org/build/tools/raw-file/default/breakpad/linux/minidump_stackwalk on one of the test slaves:

/builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk)

/builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk: /usr/lib/libstdc++.so.6: version `GLIBCXX_3.4.11' not found (required by /builds/moz2_slave/fuzzer-linux/build/build/minidump_stackwalk)

minidump_stackwalk exited with return code 1
Comment 1 User image Bob Clary [:bc:] 2010-11-08 04:48:35 PST
What version of cygwin is used on the test slaves? The one I built was with cygwin 1.7. It should be pretty easy to rebuild with CXXFLAGS="-g -O1" ./configure if the slaves are still using Cygiwn 1.5.
Comment 2 User image Ted Mielczarek [:ted.mielczarek] 2010-11-08 05:24:32 PST
This isn't Cygwin, this is Linux. We've hit this a number of times in the past, see bug 578880, bug 526868, bug 554854.
Comment 3 User image Mike Taylor [:bear] 2010-11-11 13:53:45 PST
Ted, is this an issue for running the minidump_stackwalk manually that environment variables need to be set appropriately or built for the environment he is testing in?

In otherwords I'm having trouble seeing exactly how RelEng would help solve this?
Comment 4 User image Ted Mielczarek [:ted.mielczarek] 2010-11-11 17:07:42 PST
We had this problem running test binaries (probably one of those bugs I linked in comment 2), and we needed to set LD_LIBRARY_PATH in the build environment for it to work, since we're building with a newer libstdc++ thatn what's installed on the slaves.
Comment 5 User image John Ford [:jhford] CET/CEST Berlin Time 2010-11-15 15:58:54 PST
Is this on a fedora slave?  I see that the libstdc++.so.6 library on a standard slave does have both of those strings.  Is there something more than the below command output to verify the status of this library?

[cltbld@talos-r3-fed-001 ~]$ strings /usr/lib/libstdc++.so.6 | grep GLIBCXX
GLIBCXX_3.4
GLIBCXX_3.4.1
GLIBCXX_3.4.2
GLIBCXX_3.4.3
GLIBCXX_3.4.4
GLIBCXX_3.4.5
GLIBCXX_3.4.6
GLIBCXX_3.4.7
GLIBCXX_3.4.8
GLIBCXX_3.4.9
GLIBCXX_3.4.10
GLIBCXX_3.4.11
GLIBCXX_3.4.12
GLIBCXX_3.4.13
GLIBCXX_FORCE_NEW
GLIBCXX_DEBUG_MESSAGE_LENGTH
Comment 6 User image Jesse Ruderman 2010-11-15 16:36:14 PST
I'm not sure how to get from a fuzzing log to a slave name.  Should I add something that writes the slave name to a file?
Comment 7 User image Nick Thomas (UTC+13) [:nthomas] 2010-11-15 16:59:06 PST
The fuzzing jobs run on the CentOS builders, which will have an older libstdc++ on them.
Comment 8 User image Jesse Ruderman 2010-11-17 21:31:59 PST
It would be nice to get this fixed so I can find Linux-specific crash bugs through fuzzing :)
Comment 9 User image Mike Taylor [:bear] 2010-11-18 13:12:18 PST
Assigning this to Rail.

Rail, can you look at the issue here - it seems to me that we can set the LD_LIBRARY_PATH env var to point to the GCC environment this script needs.

Can you confirm that? and if so, then make a patch to change this script to have that information?

thanks!
Comment 10 User image Rail Aliiev [:rail] ⌚️ET 2010-11-18 13:37:03 PST
IS it possible to use statically linked minidump_stackwalk? In this case we can run it on any slave.
Comment 11 User image Ted Mielczarek [:ted.mielczarek] 2010-11-18 13:49:02 PST
Yeah, there shouldn't be any problem with that, I just didn't build it that way.
Comment 12 User image Rail Aliiev [:rail] ⌚️ET 2010-11-19 05:42:05 PST
(In reply to comment #11)
> Yeah, there shouldn't be any problem with that, I just didn't build it that
> way.

Assigning to you in this case. ;)

Feel free to push it back to me when we are ready for deployment of new binaries.
Comment 13 User image Ted Mielczarek [:ted.mielczarek] 2010-12-14 14:45:51 PST
Created attachment 497624 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Here's a patch to build/tools with a statically-linked Linux minidump_stackwalk.
Comment 14 User image Ted Mielczarek [:ted.mielczarek] 2010-12-14 14:46:36 PST
Created attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Here's a patch to build/tools with a statically-linked Linux minidump_stackwalk.
Comment 15 User image Ben Hearsum (:bhearsum) 2010-12-15 17:31:36 PST
Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Welcome to stampy town.
Comment 16 User image Chris Cooper [:coop] 2011-07-08 11:23:44 PDT
Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

http://hg.mozilla.org/build/tools/rev/776728aff443
Comment 17 User image Jesse Ruderman 2011-07-19 17:27:10 PDT
Still broken, at least for 64-bit:

/builds/slave/fuzzer-linux64/build/minidump_stackwalk: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.9' not found (required by /builds/slave/fuzzer-linux64/build/minidump_stackwalk)
Comment 18 User image Jesse Ruderman 2011-07-20 02:59:53 PDT
Does seem to be working on 32-bit :)
Comment 19 User image Ted Mielczarek [:ted.mielczarek] 2011-07-20 05:35:16 PDT
Ah, crap, I apparently only fixed the 32-bit mdsw. I thought they were symlinks at one point?
Comment 20 User image Jesse Ruderman 2011-07-23 07:51:50 PDT
Having this working on 32-bit allowed me to find bug 673390 :)
Comment 21 User image Chris Cooper [:coop] 2011-07-28 12:59:22 PDT
(In reply to comment #19)
> Ah, crap, I apparently only fixed the 32-bit mdsw. I thought they were
> symlinks at one point?

Ted: does this imply that I can simply copy over the already-built statically linked version into linux64 and it will just work?
Comment 22 User image Ted Mielczarek [:ted.mielczarek] 2011-07-28 13:04:17 PDT
That ought to be fine, if it's statically linked then it shouldn't have any library dependencies to worry about, so it should work.
Comment 23 User image Chris Cooper [:coop] 2011-07-28 13:43:05 PDT
Comment on attachment 497626 [details] [diff] [review]
Statically link Linux minidump_stackwalk

Checked-in for linux64 also:

http://hg.mozilla.org/build/tools/rev/9e0852356108
Comment 24 User image Mike Hommey [:glandium] 2011-07-28 23:41:37 PDT
I don't think this works. If I recall correctly, a linux32 executable is only able to deal with linux32 at runtime. So using the linux32 executable on linux64 is not going to work.
Comment 25 User image Ted Mielczarek [:ted.mielczarek] 2011-07-29 04:23:14 PDT
It runs fine on my 64-bit Ubuntu system, FWIW.
Comment 26 User image Mike Hommey [:glandium] 2011-07-29 04:43:09 PDT
(In reply to comment #25)
> It runs fine on my 64-bit Ubuntu system, FWIW.

I must be mixing with dump_symbols.

Note You need to log in before you can comment on or make changes to this bug.