1814852 - Look for macOS __crash_info in the __DATA_DIRTY section if it can't be found in the __DATA one

Status: RESOLVED FIXED

(Toolkit :: Crash Reporting, enhancement)

Description

[Gabriele Svelto [:gsvelto]]

Apparently this happens occasionally on macOS 13. See this recent crashpad commit

Comment 1

[Steven Michaud [:smichaud] (Retired)]

The __crash_info section is in the __DATA_DIRTY segment of several modules in older versions of macOS (I checked back to macOS 11.7.3). None of them is important to Mozilla:

/System/Library/PrivateFrameworks/Safari.framework/Versions/A/Safari
/System/Library/PrivateFrameworks/AccountsDaemon.framework/Versions/A/AccountsDaemon
/System/Library/PrivateFrameworks/PhotoLibraryServices.framework/Versions/A/PhotoLibraryServices

macOS 13 added one that is important, but (if I remember right) currently isn't being searched by Mozilla breakpad code.

/usr/lib/dyld

I used the following shell command to look for these modules. The output is very noisy, but you can search through it on "__DATA_DIRTY":

find . -type f -exec ls \{\} \; -exec objdump -m --section=__DATA_DIRTY,__crash_info \{\} 2> /dev/null \;

Mozilla's Breakpad should be able to look for the __crash_info section in both __DATA and __DATA_DIRTY. And it really should be able to look in /usr/lib/dyld's __crash_info section. Over the next week or so I'll work up and test a patch that does both of these things.

Comment 2

[Steven Michaud [:smichaud] (Retired)]

Here's a more efficient version of my shell command from comment #1:

find . -type f -exec ls \{\} \; -exec objdump -m --section=__DATA_DIRTY,__crash_info \{\} 2> /dev/null \; | grep -C10 "DATA_DIRTY"

Comment 3

[Gabriele Svelto [:gsvelto]]

Thanks for the in-depth analysis Steven

Comment 4

[Steven Michaud [:smichaud] (Retired)]

I'm working on a patch for this. I'm a few days away from posting it and asking for a review.

Comment 5

[Steven Michaud [:smichaud] (Retired)]

I'm about to post my patch, so here's a short description of what it does:

dyld is macOS's "dynamic linker". It's loaded into every process, and finishes the process's initialization, after execution has been handed off to it by the kernel. It implements dlsym(), dlopen() and friends (libdyld.dylib is just a thin wrapper around this functionality). And it contains a structure (dyld_all_image_infos) with important information on the current process -- for example a list of modules that have been loaded into that process. But, oddly, this list doesn't contain dyld itself. And for this reason Breakpad doesn't include information on it in its minidumps: dyld isn't in the mindump's list of modules, and it isn't searched for __crash_info.

Furthermore, as pointed out above, some frameworks/dylibs have a __crash_info section in the __DATA_DIRTY segment. But current Breakpad code only looks for __crash_info in the __DATA segment (its normal location).

My patch fixes these problems. With its changes, Breakpad (on macOS) includes dyld in every minidump's list of modules. It searches the dyld module for __crash_info. And whenever a __crash_info section isn't found in a module's __DATA segment, it also looks in the __DATA_DIRTY segment, if present.

I've made a tryserver build and tested with it:

https://treeherder.mozilla.org/jobs?repo=try&revision=f5091d91ed2415fed10616a9c08995dd995a4114
https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/E_-J__jDSIaDoXMhZIrJVQ/runs/0/artifacts/public/build/target.dmg

It does everything I expected, plus one thing I didn't: Now that dyld is included in each minidump's list of modules, the stackwalker symbolicates crash stack entries that come from dyld.

Here are some crash reports that I generated using a HookCase hook library:

macOS 12.6.3:
bp-cf32e134-e13a-43fa-aefc-5d35c0230217
bp-55f49206-f1c8-4d94-9dfd-de8fd0230217

macOS 13.2:
bp-9e63e2ad-859d-4081-b651-f9cab0230217
bp-4518d833-4bd8-4317-ac66-7eb2a0230217

Comment 6

[Steven Michaud [:smichaud] (Retired)]

Attachment: Bug 1814852 - Add support for dyld and __DATA_DIRTY,__crash_info to Breakpad. r=gsvelto

Comment 7

[Steven Michaud [:smichaud] (Retired)]

Attachment: HookCase hook library I tested with

Here's the HookCase hook library I tested with, as a diff on https://github.com/steven-michaud/HookCase/blob/master/HookLibraryTemplate/hook.mm.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/929ba205f9d5
Add support for dyld and __DATA_DIRTY,__crash_info to Breakpad. r=gsvelto

Comment 9

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/929ba205f9d5