445090 - Thunderbird Linux buildbot sometimes crashing in dump_syms [malloc(): memory corruption]

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Philippe M. Chiasson (:gozer)]

Attachment: Relevant snippet of the actual log

+++ This bug was initially created as a clone of Bug #411171 +++

I also recently started noticing similar, yet different failures in dump_syms on the new Linux Thunderbird Buildbot:

dump_syms: [mozilla]/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:392: void<unnamed>::AddIncludedFiles(<unnamed>::SymbolInfo*, const Elf32_Shdr*): Assertion `line_info.source_name_index > 0' failed.
dump_syms: [mozilla]/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:392: void<unnamed>::AddIncludedFiles(<unnamed>::SymbolInfo*, const Elf32_Shdr*): Assertion `line_info.source_name_index > 0' failed.
*** glibc detected *** [mozilla]/objdir/dist/host/bin/dump_syms: malloc(): memory corruption: 0x08f71f30 ***

Comment 1

[Philippe M. Chiasson (:gozer)]

Removing the dependency automatically introduced when cloning this bug.

Comment 2

[Robert Kaiser]

gozer, it probably would be better if you name this "buildbot" instead of "tinderbox" in the summary, as those two words are used for different client-side builder programs, though both report to the tinderbox server in our config.

That said, I'm sseing the same crash on two places in the SeaMonkey calemaisu-test nightly builder, and AFAIK I also saw them for Firefox nightly builders.

Comment 3

[Philippe M. Chiasson (:gozer)]

(In reply to comment #2)
> gozer, it probably would be better if you name this "buildbot" instead of
> "tinderbox" in the summary,

renamed

> That said, I'm sseing the same crash on two places in the SeaMonkey
> calemaisu-test nightly builder, and AFAIK I also saw them for Firefox nightly
> builders.

Do they happen all the time, or are they intermittent like I am seeing ? Links to relevant logs/bugids would be appreciated

Comment 4

[Robert Kaiser]

(In reply to comment #3)
> Do they happen all the time, or are they intermittent like I am seeing ? Links
> to relevant logs/bugids would be appreciated

I've seen them in all FF logs I looked into ("Firefox" waterfall, "Linux mozilla-central nightly" column, e.g. <http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1216026000.1216028763.30376.gz&fulltext=1>) and in the few SeaMonkey logs I looked into as well ("Linux sm-hg-test nightly" column on "SeaMonkey-Ports" only erratically produces builds as it needs forcing, as I didn't turn on any scheduler, but <http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey-Ports/1215978061.1215983216.26479.gz&fulltext=1> is one of the logs). Search for "dump_syms:" returns two such crashes consistently for those.

Comment 5

[(not currently active) Ted Mielczarek]

I forwarded this to Liu Li (who wrote the Linux dump_syms implementation) and he thinks he knows what the problem is.

Comment 6

[(not currently active) Ted Mielczarek]

Li says there's a patch here: http://code.google.com/p/google-breakpad/issues/detail?id=259 that might fix this. Anyone care to apply it somewhere and test?

Comment 7

[Philippe M. Chiasson (:gozer)]

I've applied it in my own build box. Without it, I could get the segfault every single time I'd run make buildsymbols. After applying the patch, running 'make buildsymbols' in a loop failed to crash it even once. Sounds like a valid band-aid to this problem.

Comment 8

[Philippe M. Chiasson (:gozer)]

Any chance of landing that patch ?

For the record, this is obviously still hapenning, and this time, I can show the full failing stack trace:

$>  gdb ./mozilla/dist/host/bin/dump_syms
[...]
(gdb) run mozilla/dist/bin/thunderbird-bin
Starting program: [comm-central]/objdir/mozilla/dist/host/bin/dump_syms mozilla/dist/bin/thunderbird-bin

Program received signal SIGSEGV, Segmentation fault.
0x0804eda3 in (anonymous namespace)::AddIncludedFiles (symbols=0xbf9dc37c, stabstr_section=0xb7fbbc7c)
    at [comm-central]/mozilla/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:392
392             assert(line_info.source_name_index > 0);
(gdb) bt
#0  0x0804eda3 in (anonymous namespace)::AddIncludedFiles (symbols=0xbf9dc37c, stabstr_section=0xb7fbbc7c)
    at [comm-central]/mozilla/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:392
#1  0x080509df in (anonymous namespace)::LoadSymbols (stab_section=0xb7fbbc54, stabstr_section=0xb7fbbc7c, loading_addr=134512640,
    symbols=0xbf9dc37c)
    at [comm-central]/mozilla/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:544
#2  0x08050ad5 in (anonymous namespace)::LoadSymbols (elf_header=0xb7f9b000, symbols=0xbf9dc37c)
    at [comm-central]/mozilla/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:569
#3  0x080514ba in google_breakpad::DumpSymbols::WriteSymbolFile (this=0xbf9dc3e3, obj_file=@0xbf9dc3e4, sym_fd=1)
    at [comm-central]/mozilla/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:743
#4  0x08048d3a in main (argc=-1080179608, argv=0x561390)
    at [comm-central]/mozilla/toolkit/crashreporter/google-breakpad/src/tools/linux/dump_syms/dump_syms.cc:46

Comment 9

[(not currently active) Ted Mielczarek]

I'll get this landed upstream and get it into our tree (at some point).

Comment 10

[Wayne Mery (:wsmwk)]

bump

Comment 11

[Wayne Mery (:wsmwk)]

(In reply to comment #9)
> I'll get this landed upstream and get it into our tree (at some point).

Ted, possible to land?

I think we had Mac syms off and on, but for some weeks now there are zero.

Comment 12

[(not currently active) Ted Mielczarek]

Ok, I landed this upstream:
http://code.google.com/p/google-breakpad/source/detail?r=296

I'm going to sync to Breakpad trunk very soon.

Comment 13

[Philippe M. Chiasson (:gozer)]

Just realized that patch is linux-only. Is it possible that the osx version suffers from a similar problem. Looking at bug 466385, it's failing to generate symbols, for a large binary (thunderbird-bin) and is failing deep inside malloc(), so maybe related ?

Comment 14

[(not currently active) Ted Mielczarek]

Let's re-evaluate the mac problems once I land DWARF support (bug 421534).

Comment 15

[Philippe M. Chiasson (:gozer)]

It's happening to Firefox too, it would seem.

From: http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1228212000.1228215346.15082.gz&fulltext=1

dump_syms: /builds/moz2_slave/mozilla-central-linux-nightly/build/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:392: void<unnamed>::AddIncludedFiles(<unnamed>::SymbolInfo*, const Elf32_Shdr*): Assertion `line_info.source_name_index > 0' failed.

Comment 16

[(not currently active) Ted Mielczarek]

That's different, that's bug 415316.

Comment 17

[(not currently active) Ted Mielczarek]

Of course you guys had to go and follow the 1.9.1 branch, so you didn't get the fix yet. ;-)

I'm going to try to land bug 421534 on the 1.9.1 branch after it bakes for a while on trunk.

Comment 18

[(not currently active) Ted Mielczarek]

Should be fixed on both m-c and 1.9.1 by the landing of bug 465329.