714129 - Breakpad dump_syms does not support DWARF 4 attribute forms, produced by GCC 4.6.2, causing dump_syms crash

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Bob Clary [:bc] (inactive)]

1. Build Firefox (Beta|Aurora|Nightly) on Fedora 16 with stock gcc 4.6.2 with -gdwarf-2
2. dump_syms libxul.so


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Unhandled form typeUnhandled form type
Program received signal SIGSEGV, Segmentation fault.
0x0806b049 in dwarf2reader::ByteReader::ReadUnsignedLEB128(char const*, unsigned int*) const ()
(gdb) bt
#0  0x0806b049 in dwarf2reader::ByteReader::ReadUnsignedLEB128(char const*, unsigned int*) const ()
#1  0x0806ddad in dwarf2reader::CompilationUnit::ProcessDIEs() ()
#2  0x0806d091 in dwarf2reader::CompilationUnit::Start() ()
#3  0x0804b3b8 in LoadDwarf (dwarf_filename=..., elf_header=0xa3899000, big_endian=false, module=0xbffff064)
    at /work/mozilla/builds/beta/mozilla/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:282
#4  0x0804c068 in LoadSymbols (obj_file=..., big_endian=false, elf_header=0xa3899000, 
    read_gnu_debug_link=true, info=0xbffff0b8, module=0xbffff064)
    at /work/mozilla/builds/beta/mozilla/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:559
#5  0x0804c782 in google_breakpad::WriteSymbolFile (obj_file=..., debug_dir=..., sym_file=0x81c3ce0)
    at /work/mozilla/builds/beta/mozilla/toolkit/crashreporter/google-breakpad/src/common/linux/dump_symbols.cc:712
#6  0x0804a054 in main (argc=2, argv=0xbffff1f4)
    at /work/mozilla/builds/beta/mozilla/toolkit/crashreporter/google-breakpad/src/tools/linux/dump_syms/dump_syms.cc:49

Adding -gstrict-dwarf did not help. Note you will also see bug 563791.

Comment 1

[Jim Blandy :jimb]

Could you post the output from running  'readelf -WS' on a libxul.so that crashes dump_syms? Just a hunch.

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: readelf output

Comment 3

[Jim Blandy :jimb]

Any chance I could get a copy of the dump_syms and the libxul.so (don't attach them to the bug; that'd be ridiculous)? The dump_syms was build with debugging symbols, right?

Comment 4

[Bob Clary [:bc] (inactive)]

I clobbered that build and rebuilt with gcc45 but did save the libxul.so but it is huge... 378M. But yes, I think the dump_syms was built debug as it was built along with firefox. I can rebuild again but I don't know where I can put libxul.so. I can do this in the phx colo on a fedora 16 box there. Do you have access to the mpt vpn?

Comment 5

[Jim Blandy :jimb]

I've accessed the MPT VPN before, but it took a lot of setup and messing around with VPN software on Linux that I don't really understand. I don't need libxul.so sources; I just need any ELF file that makes dump_syms crash.

Comment 6

[Jim Blandy :jimb]

bc and I have worked together to get me the files I needed, and I can reproduce this on my machine.

Comment 7

[Jim Blandy :jimb]

It turns out that not all the compilation units for dump_syms were built with debugging info, so I had to poke around in the assembly a bit, but I'm pretty sure about the diagnosis.

The problem here is that GCC is producing DWARF 4 debugging information, which includes some attribute forms that dump_syms does not recognize. dump_syms needs to be updated to handle the new attribute forms in DWARF 4. Just being able to skip them safely is not very much work. It's possible the quality of the output won't even be degraded, although it might.

Crashing on new input forms seem like an egregious bug, but until DWARF 4, DWARF was remarkably successful at allowing older readers to skip information they didn't understand, largely by leaving the vocabulary of attribute forms unchanged since DWARF 2. Until DWARF 4, trying to read compilation units with newer DWARF format numbers than expected would have been the smart thing to do.

Details:

Before crashing, dump_syms prints:

Unhandled form typeUnhandled form type

These (missing newlines and all) are from dwarf2reader::CompilationUnit::ProcessAttribute, which returns NULL when it sees an attribute form it doesn't know. Its callers don't check for that, and the next call to ReadUnsignedLEB128 tries to read an abbreviation number from address zero.

(Just for the record, the DIE parser in dwarf2reader.cc is not code I wrote, and is mostly untested.)

Comment 8

[Jim Blandy :jimb]

I'm a little surprised about this, though: while there are definitely DWARF 4 compilation units in that binary, GCC seems to promise not to generate DWARF 4 unless specifically permitted to do so, using the -gdwarf-4 flag.

You don't have the flags available for that build, do you?

Comment 9

[Bob Clary [:bc] (inactive)]

I have 

ac_add_options --enable-debug-symbols="-gdwarf-2"
export CFLAGS="-gdwarf-2"
export CXXFLAGS="-gdwarf-2"

Comment 10

[Bob Clary [:bc] (inactive)]

http://old.nabble.com/GCC-4.6.1-emits-discriminators-in-DWARF2-mode-td32689907.html

iirc during my attempts to work around the error i did try -gdwarf-strict but it didn't help.

Comment 11

[Jim Blandy :jimb]

I'm pretty sure GCC will not produce DWARF 4 unless invited to.

It seems like it's the mozilla/security/nss/lib/crmf directory that's getting built with DWARF 4. Can you see any -gdwarf-X options being used there?

Comment 12

[Jim Blandy :jimb]

Here are the main source files for the compilation units that have DWARF 4 debug info:

crmfenc.c	
crmftmpl.c	
crmfreq.c	
crmfpop.c	
crmfget.c	
crmfcont.c	
encutil.c	
respcli.c	
respcmn.c	
asn1cmn.c	

All are in security/nss/lib/crmf.

Comment 13

[Bob Clary [:bc] (inactive)]

No, but it appears to me that my -gdwarf-2 option is not being used there. Instead the compilation lines just contain -g. Funnily, when I just did which gcc, I got /usr/lib/ccache/gcc. It is 4.6.2-1 from redhat which is the same as /usr/bin/gcc.

Comment 14

[Jim Blandy :jimb]

Perhaps Fedora has local patches which enable DWARF 4. I'm looking at the G++ 4.6.2 release, and there's really only one place that dwarf_level gets assigned to; its default is 2; and everything in dwarf2out.c checks dwarf_level.

But that's neither here nor there. The only thing in DWARF 4 that would be a lot of work to support is DW_FORM_ref_sig8, and GCC's not generating that. (In my sources.) So the patch is straightforward.

Comment 15

[Jim Blandy :jimb]

Attachment: Stopgap fix: Don't try to read .debug_info compilation units using a version of DWARF we don't understand.

The right fix is to extend Breakpad's DWARF parser to understand DWARF 4's
new attribute forms, but since very few of our compilation units actually
contain DWARF 4 data, this should let us get running again until the right
fix is done.

Comment 16

[(not currently active) Ted Mielczarek]

Comment on attachment 588595 [details] [diff] [review]
Stopgap fix: Don't try to read .debug_info compilation units using a version of DWARF we don't understand.

Review of attachment 588595 [details] [diff] [review]:
-----------------------------------------------------------------

Seems fine. If we're only breaking in NSS, perhaps we should file a NSS bug on adding whatever options we need to make this specific GCC not produce DWARF 4? Or is it a GCC bug (perhaps only with Fedora's GCC?) that it's producing DWARF 4 in this situation?

Comment 17

[(not currently active) Ted Mielczarek]

Also, feel free to land this both in mozilla-central and upstream in Breakpad at the same time.

Comment 18

[Jim Blandy :jimb]

I've posted some patches upstream that actually go ahead and add support for the DWARF 4 attribute forms.

Comment 19

[Jim Blandy :jimb]

... and here's a link:
https://code.google.com/p/google-breakpad/issues/detail?id=461&sort=-id

Comment 20

[Jim Blandy :jimb]

Attachment: Add partial unit tests for dwarf2reader::CompilationUnit. (r=ted.mielczarek, upstream)

This is really incomplete --- it's just what's needed to get started
testing support for the DWARF 4 attribute forms.

Comment 21

[Jim Blandy :jimb]

Attachment: Breakpad DWARF support: Remove extraneous breaks following returns in dwarf2reader.cc. (r=ted.mielczarek, upstream)

Comment 22

[Jim Blandy :jimb]

Attachment: Breakpad DWARF: Add support for DWARF 4 attribute forms. (r=ted.mielczarek, upstream)

This patch allows Breakpad's DWARF reader to at least read or skip
attributes using the new forms defined in version 4 of the DWARF
specification, instead of crashing.

Attributes encoded using DW_FORM_flag_present, DW_FORM_sec_offset, and
DW_FORM_exprloc should work fine now. However, compilation units using
DW_FORM_ref_sig8 to refer to types in .debug_types will need further work
to support. (GCC 4.6.2 does not emit .debug_types sections.)

Specifically:

- dwarf2reader::DwarfForm gets new values.
- dwarf2reader::Dwarf2Handler and dwarf2reader::DIEHandler get new handler
  methods, named ProcessAttributeSignature, for DW_FORM_ref_sig8 attributes.
- dwarf2reader::CompilationUnit reads DW_FORM_ref_sig8 attributes, and
  passes them to ProcessAttributeSignature. It also gets support for
  DW_FORM_sec_offset, DW_FORM_exprloc, and DW_FORM_flag_present, using the
  existing appropriate ProcessAttribute* methods.
- dwarf2reader::DIEDispatcher passes through ProcessAttributeSignature
  attributes to its DIEHandler.
- Unit tests are updated.

Comment 23

[Jim Blandy :jimb]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4d1e22e8f50f

Comment 24

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/4d1e22e8f50f