830952 - Mac crash reports are missing source file links

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, defect)

Description

[Jesse Ruderman]

Mac crash reports are missing source file links.  I guess Socorro has filenames but not the rest of the srcdir-relative paths.

Example: bp-08da0359-f8e6-419c-ab9d-6f4cc2121219

Is it possible this has been broken since we switched to Clang (bug 733905)?

Comment 1

[Rafael Ávila de Espíndola (:espindola) (not reading bugmail)]

Maybe. Where is the code that produces the links?

Comment 2

[(not currently active) Ted Mielczarek]

We post-process the output of dump_syms in the symbolstore.py script:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py#588

It takes FILE records, looks up the filename to see if it's in a known VCS, and changes the filename to include references to the repository path and changeset if so. Looking at the "Raw Dump" tab of the crash from comment 0, you can see that this clearly isn't working:
0|1|XUL|mozilla::css::RestyleTracker::DoProcessRestyles()|../../../../layout/base/RestyleTracker.cpp|248|0x4

If I had to guess, I'd say this is because Breakpad's DWARF parser doesn't support the compilation directory attribute, and clang isn't putting absolute paths in the source file attributes (presumably GCC 4.2 was). There's a patch that's been sitting upstream for this for a while, we should poke Ryan Sleevi to get that landed:
https://breakpad.appspot.com/385001/

Comment 3

[Ryan Sleevi]

I've updated the codereview with the latest and greatest code that should work for you.

I don't know if Firefox is using anything wired up through XCode (I didn't think so), but in the event you are, http://openradar.appspot.com/13031540 has a chance of biting you.

Comment 4

[(not currently active) Ted Mielczarek]

Thanks Ryan! I'd love to get that landed and merged into our tree, I suspect it's the underlying issue here. Thanks for that pointer, but yeah, we just invoke clang via make.

Comment 5

[Ryan Sleevi]

Ted: Would you mind stamping that review then, if it looks good? And/or confirming that it would work for you?

dwarfdump --debug-info [file].dSYM | grep comp_dir
dwarfdump --debug-line [file].dSYM | grep 'include_dir\|file_names'

Comment 6

[(not currently active) Ted Mielczarek]

I'm not exactly sure what the expected output of those dwarfdump commands should be. Testing on a firefox.dSYM from a recent nightly build, the former produces quite a few lines, and the latter produces nothing.

Comment 7

[Ryan Sleevi]

Apologies. It looks like dwarfdump isn't smart enough to handle a naked .dSYM, you need to dig into the <app>.dSYM/Contents/Resources/DWARF/<app>

Instead of a full .dSYM, you can always use a .o file from a local build, which is often easier and quicker.

A sample output from my dummy executable on the aforementioned bug (here, I'm using a .o)

acsleevi:development sleevi$ dwarfdump /Users/sleevi/development/xcode-bug/dummy2/dir3/DerivedData/dummy2/Build/Intermediates/dummy2.build/Debug/dummy2.build/Objects-normal/x86_64/main.o | grep comp
0x0000000b: TAG_compile_unit [1] *
             AT_comp_dir( "/Users/sleevi/development" )

macsleevi:development sleevi$ dwarfdump --debug-line /Users/sleevi/development/xcode-bug/dummy2/dir3/DerivedData/dummy2/Build/Intermediates/dummy2.build/Debug/dummy2.build/Objects-normal/x86_64/main.o | grep 'include_dir\|file_names'
include_directories[  1] = '/Users/sleevi/development/xcode-bug/dummy2/dir3/dir4'
include_directories[  2] = '/Applications/Xcode4.5.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/usr/include/c++/4.2.1'
include_directories[  3] = '../include/foo/../bar'
file_names[  1]    1 0x00000000 0x00000000 main.cc
file_names[  2]    2 0x00000000 0x00000000 limits
file_names[  3]    3 0x00000000 0x00000000 foo.h


Or you can just test a dump_syms with my change patched in, and see if the FILE lines in the resulting .sym appear sane.

Comment 8

[(not currently active) Ted Mielczarek]

Sorry, I got tied up in some other things and didn't get a chance to test this until just now. Building dump_syms with your patch applied gives me sensible results when dumping symbols from the same firefox.dSYM above. Thanks!

Comment 9

[(not currently active) Ted Mielczarek]

I landed Ryan's patch upstream:
http://code.google.com/p/google-breakpad/source/detail?r=1106

We should be able to update Breakpad to pick up this fix.

Comment 10

[(not currently active) Ted Mielczarek]

Should be fixed by bug 835456.

Comment 11

[(not currently active) Ted Mielczarek]

This seems mostly fixed:
https://crash-stats.mozilla.com/report/index/bp-2a0b0dc2-296c-436a-b042-0aed52130129

We don't have links for jemalloc.c there, but for everything else we have source links.