Closed Bug 1510574 Opened 10 months ago Closed 8 months ago
Linux32 crash symbols missing at least in mochitests
Bug 1510574 - Remove shared memory segments from generated minidumps to cut down on the number of modules
47 bytes, text/x-phabricator-request
|Details | Review|
Duplicate of this bug: 1510546
Can you take a look why the symbols are missing here, Gabriele? Thank you.
Duplicate of this bug: 1446206
I grabbed a minidump from the failing ones and the issue there was fairly silly. Breakpad's minidump-reading logic fails if the number of modules in the modules list exceeds a certain number: https://searchfox.org/mozilla-central/rev/0859e6b10fb901875c80de8f8fc33cbb77b2505e/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2725 This can be overridden but minidump_stackwalk sticks with the default which is 2048: https://searchfox.org/mozilla-central/rev/0859e6b10fb901875c80de8f8fc33cbb77b2505e/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#3861 The minidump I looked at had 5800+ modules (mostly because of deleted mappings). I bumped up the limit locally and the minidump was processed just fine, symbolicated stack and all. It took ~7 seconds on my machine which is a fair bit longer than usual but nothing dramatic. Ted, do you think we can raise this limit? minidump_dump already sets it to MAX_UINT32 so maybe we can go a bit higher than 2048 without suffering adverse effects.
Flags: needinfo?(gsvelto) → needinfo?(ted)
Assignee: nobody → gsvelto
Status: NEW → ASSIGNED
you'll need using google_breakpad::MinidumpModuleList; as well. btw, this fixes a long standing issue for me though I also included the change to minidump.cc as well though not sure it is required.
May have spoken too soon. Partially fixed it appears.
Summary: Linux debug crash symbols missing at least in mochitests → Linux32 crash symbols missing at least in mochitests
I commented in Phabricator, but we don't currently build the in-tree code for use in automation. glandium updated the (very old) binaries we had been using in bug 1512706, but punted on a more complete solution because it got complicated. It looked like he might fix bug 1391408 though.
Depends on: 1391408
I will, but I also won't go further to hook it up for test jobs to download it from the toolchain tasks.
I was wondering, would it make sense to build minidump_stackwalk as an host binary like we do with dump_syms instead of taking it from outside of mozilla-central?
That's essentially what I have WIP patches for (except it's not /exactly/ building them as host binaries)
Would this affect Linux64 too? I've been running into this on my Linux64 try pushes for at least a week, and it's a huge pain... If this bug doesn't cover Linux64, I'm happy to file a separate one to track that.
(In reply to Boris Zbarsky [:bzbarsky, bz on IRC] from comment #13) > Would this affect Linux64 too? I've been running into this on my Linux64 > try pushes for at least a week, and it's a huge pain... Yes, it would affect all platforms.
(In reply to Gabriele Svelto [:gsvelto] from comment #4) > The minidump I looked at had 5800+ modules (mostly because of deleted mappings). Is there a way to convince breakpad to ignore non-executable mappings? If it weren't picking up shared memory and data files, that number would probably be a lot smaller. (Shared memory segments on Linux currently show up as deleted files, because they are, but I've also seen breakpad pick up memfd mappings created by system libraries so bug 1440203 probably won't help here.)
The Linux dump writing code does all sorts of fiddling with the list of mappings, so we could absolutely filter out stuff that's not useful: https://dxr.mozilla.org/mozilla-central/rev/c2593a3058afdfeaac5c990e18794ee8257afe99/toolkit/crashreporter/breakpad-client/linux/minidump_writer/linux_dumper.cc#575 We do stuff /proc/self/maps into the minidump verbatim (it's hard to get at without running minidump_dump, but it's there), so I don't think we'd lose much by doing that.
Pushed by email@example.com: https://hg.mozilla.org/integration/autoland/rev/30fd54bb8089 Remove shared memory segments from generated minidumps to cut down on the number of modules r=ted
You need to log in before you can comment on or make changes to this bug.