1544881 - dump_syms crashes processing libxul.so compiled with GCC 8.3.0

Status: VERIFIED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I tried doing ./mach buildsymbols on a build compiled with GCC 8.3.0 (to contrast its debug info with Clang), and it didn't include debug symbols for libxul.so. This is because dump_syms is failing an assertion:

assert(!line || current < line->address || within(*line, current));

Stack:

#4  0x0000555555578150 in google_breakpad::DwarfCUToModule::AssignLinesToFunctions (this=0x7fffffffda00, files=...) at /usr/include/c++/8/bits/stl_vector.h:565
#5  0x000055555557822d in google_breakpad::DwarfCUToModule::Finish (this=0x7fffffffda00) at /home/jld/src/gecko-dev/toolkit/crashreporter/google-breakpad/src/common/dwarf_cu_to_module.cc:1521
#6  0x0000555555580a72 in dwarf2reader::DIEDispatcher::EndDIE (this=0x7fffffffdb10, offset=<optimized out>) at /usr/include/c++/8/bits/stl_deque.h:255
#7  0x0000555555585898 in dwarf2reader::CompilationUnit::ProcessDIEs (this=0x7fffffffdbe0) at /home/jld/src/gecko-dev/toolkit/crashreporter/google-breakpad/src/common/dwarf/dwarf2reader.cc:583
#8  0x00005555555880fc in dwarf2reader::CompilationUnit::Start (this=this@entry=0x7fffffffdbe0) at /home/jld/src/gecko-dev/toolkit/crashreporter/google-breakpad/src/common/dwarf/dwarf2reader.cc:347
#9  0x000055555556a8ac in (anonymous namespace)::LoadDwarf<google_breakpad::ElfClass64> (module=0x55555565e220, handle_inter_cu_refs=<optimized out>, big_endian=false, elf_header=0x7fff62c59000, 
    dwarf_filename="obj-x86_64-pc-linux-gnu/dist/bin/libxul.so") at /usr/include/c++/8/bits/stl_deque.h:149

And this is the input to that assertion ($r12 is line here):

(rr) p/x *(Module::Line*)$r12
$5 = {address = 0x166823f, size = 0x0, file = 0x563fb0a968e0, number = 0xa5}
(rr) p/x current
$6 = 0x166823f
(rr) p ((Module::Line*)$r12)->file[0]
$11 = {name = "/home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/dist/include/mozilla/Assertions.h", source_id = -1}

I don't know this code or DWARF well, but I'm suspicious about that zero size.

Here's what addr2line thinks about that address:

$ addr2line -Cfie obj-x86_64-pc-linux-gnu/dist/bin/libxul.so 0x166823f
MOZ_ReportAssertionFailure
/home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/dist/include/mozilla/Assertions.h:159

And readelf --debug-dump=decodedline:

CU: /home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/dist/include/mozilla/Assertions.h:
File name                            Line number    Starting address    View
Assertions.h                                 159           0x166823f
Assertions.h                                 165           0x166823f       1

/usr/include/x86_64-linux-gnu/bits/stdio2.h:
stdio2.h                                      98           0x166823f       2
stdio2.h                                     100           0x166823f       3

/home/jld/src/obj.gecko-dev/obj-x86_64-pc-linux-gnu/dist/include/mozilla/Assertions.h:
Assertions.h                                 159           0x166823f       4

And readelf --debug-dump=rawline:

 Line Number Statements:
  [0x00013434]  Set column to 80
  [0x00013436]  Extended opcode 2: set Address to 0x166823f
  [0x00013441]  Advance Line by 158 to 159
  [0x00013444]  Copy
  [0x00013445]  Set column to 3
  [0x00013447]  Special opcode 11: advance Address by 0 to 0x166823f and Line by 6 to 165 (view 1)
  [0x00013448]  Set File Name to entry 2 in the File Name Table
  [0x0001344a]  Set column to 1
  [0x0001344c]  Advance Line by -67 to 98
  [0x0001344f]  Copy (view 2)
  [0x00013450]  Set column to 3
  [0x00013452]  Special opcode 7: advance Address by 0 to 0x166823f and Line by 2 to 100 (view 3)
  [0x00013453]  Set File Name to entry 1 in the File Name Table
  [0x00013455]  Set column to 80
  [0x00013457]  Set is_stmt to 0
  [0x00013458]  Advance Line by 59 to 159
  [0x0001345a]  Copy (view 4)

I also found this object, referencing line 159 with the same address and a non-zero size:

(rr) p/x ((Module::Line*)$r12)[1]
$9 = {address = 0x166823f, size = 0xa, file = 0x563fb0a968e0, number = 0x9f}

Could this be fallout from bug 524410?

Comment 1

[Gabriele Svelto [:gsvelto] (PTO)]

I tried this yesterday on a Fedora 29 install with gcc 8.3.0 but didn't encounter any crashes with a debug. Did you try it on an opt build? This is most likely fallout from bug 524410 and should be easy to fix if I can repro. Additionally since it's really dependent on the debug information it might not even be gcc-specific.

Comment 2

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

No, this is a debug build. I can send you a libxul.so that reproduces it if that would help.

Note that ./mach buildsymbols ignores the crash and silently omits the symbols from the .zip; I'm told there's already a bug on that but I didn't find it when I was filing this.

Comment 3

[Gabriele Svelto [:gsvelto] (PTO)]

Thanks, I just managed to reproduce this on a different machine (Gentoo, with GCC 8.3.0) and a stack of WIP patches on top of revision 5d2aa419a72c. I'll set aside this libxul.so and inspect it on Monday.

Comment 4

[Gabriele Svelto [:gsvelto] (PTO)]

I believe I found the root cause of this issue: before bug 524410 we read the lines from the DWARF information and explicitly skipped line entries that had a zero size. After bug 524410 we're now generating new line entries by merging ranges using information on the inlined functions: this can create entries that are zero in length thus breaking one of the invariants required by the assign-lines-to-functions logic.

Comment 5

[Gabriele Svelto [:gsvelto] (PTO)]

The issue is actually simpler than what I described in comment 4: the zero-sized entry you saw in comment 0 belongs to the inlined ranges and we were adding it to our ranges list instead of ignoring it. Since all the downstream logic assumes that ranges are non-empty we ended up generating a zero-sized line entry and ultimately assert()'ing (which is what I saw in comment 4). The zerp-sized entry we generated confused me. Fortunately the fix is really simple: just ignore empty ranges in inlined ranges.

Comment 6

[Gabriele Svelto [:gsvelto] (PTO)]

Attachment: Bug 1544881 - Ignore empty ranges when parsing DW_TAG_inlined_subroutine tags as they break the assumptions required during line assignment

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9bbe07750de9
Ignore empty ranges when parsing DW_TAG_inlined_subroutine tags as they break the assumptions required during line assignment r=froydnj

Comment 8

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/9bbe07750de9