474863 - Invalid minidump being produced without a thread list (can't be processed by minidump_stackwalk, stackwalk.sh failed)

Status: RESOLVED INCOMPLETE

(Toolkit :: Crash Reporting, defect)

Description

[Daniel Holbert [:dholbert]]

I just submitted a crash report to the crash-stats server, but it's not processing it -- when I try to load its URL...
http://crash-stats.mozilla.com/report/pending/249e42c9-fc6c-4835-8e95-4ea192090122
... I get a page with these contents:

-------------------
Your report is being processed
    * We've given your report priority
    * Your report should be ready in a minute
    * This page will refresh and display your report's status
    * When your report has been processed, we will redirect you to your report

Queue Info
ID
    249e42c9-fc6c-4835-8e95-4ea192090122
Time Queued
    2009-01-22 13:51:12.196595
Time Started
    2009-01-22 14:20:19.773969
Message
    <type 'instance'>:/home/processor/stackwalk/bin/stackwalk.sh failed with return code 1 when processing dump 249e42c9-fc6c-4835-8e95-4ea192090122
-------------------

Note in particular the final line there re "stackwalk.sh failed" -- looks like something's broken there.

Also, the status page http://crash-stats.mozilla.com/status says the crash server's mood is "Deathly" with 1250 jobs waiting -- I'm not sure if that's related.

(See also bug 459397, filed a few months back, and bug 472775, filed a few weeks back.  They both mention a similar stackwalk.sh failure.)

Comment 1

[K Lars Lohn [:lars] [:klohn]]

I've retrieved this crash dump from the production server and sent it through the my development server.  The actual dump file is huge at 129MB. The processor made these comments:

/home/processor/stackwalk/bin/minidump_stackwalk returned no header lines for reportid: 2028
no thread was identified as the cause of the crash
No signature could be created because we don't know which thread crashed
/home/processor/stackwalk/bin/minidump_stackwalk returned no frame lines for reportid: 2028
/home/processor/stackwalk/bin/minidump_stackwalk failed with return code 1 when processing dump

Comment 2

[K Lars Lohn [:lars] [:klohn]]

clearly, minidump_stackwalk doesn't like this crash dump.  Ted, if you're interested in exploring this further, you can find the dump and json files at: http://people.mozilla.com/~lars/breakpad

Comment 3

[(not currently active) Ted Mielczarek]

I don't have time at the moment to try this out, but could you capture the raw output (stdout+stderr) of running minidump_stackwalk on this dump? Also, 129Mb??!! That's ridiculously big. I've only ever seen dumps of up to 1Mb in cases of stack overflow.

Comment 4

[K Lars Lohn [:lars] [:klohn]]

I've run minidump_stackwalk manually and captured the output.  There was nothing on stdout.  All output came from stderr: http://people.mozilla.com/~lars/breakpad/249e42c9-fc6c-4835-8e95-4ea192090122.stderr

Comment 5

[(not currently active) Ted Mielczarek]

Thanks:
2009-01-28 07:57:45: minidump.cc:3635: INFO: GetStream: type 3 not present
2009-01-28 07:57:45: minidump_processor.cc:101: ERROR: Minidump /home/lars/breakpad/oldStyleTestData/testdata.diditfail/249e42c9-fc6c-4835-8e95-4ea192090122.dump has no thread list

That's the relevant output. The minidump doesn't have a list of threads in the process, so it can't actually be processed. Sounds like it's a broken minidump.
I see this is a dump from Linux. On Linux, we do a lot of things in the crashed process to collect all the necessary info. It's a bit fragile, as obviously process has already crashed, so any time you touch memory or anything you run the risk of having things fail again.

Comment 6

[K Lars Lohn [:lars] [:klohn]]

This all means:
 1: Socorro worked normally for this crash (after the 1/29/09 upgrade, Socorro will report the same messages that I echoed in Comment #1 above and then retain the report)
 2: minidump_stackwalk behaved within specs, too.  There's no much it can do with a bad dump.
 3: the problem lies with breakpad on the client or with the client itself.

Options for actions to be taken:
 1: live with it - it's not always possible to generate a proper dump file from a crashed program
 2: try to repeat the trouble and adjust/fix the breakpad client libraries to prevent this from happening.

Ted, a fear you're the one that will have to decide the disposition of this bug.

Comment 7

[(not currently active) Ted Mielczarek]

-> Breakpad Integration, anyway. Socorro is just dealing with a broken dump.

Comment 8

[Daniel Holbert [:dholbert]]

(In reply to comment #6)
> Options for actions to be taken:
>  2: try to repeat the trouble and adjust/fix the breakpad client libraries to
> prevent this from happening.

FWIW: As the originator of the crash report in question, I have no idea how to reproduce the problem... IIRC, it was just a random crash, and I don't remember much about what I did (if anything) to trigger it.

Comment 9

[(not currently active) Ted Mielczarek]

Yeah, that's a bummer. If we find a reproducible crash that triggers this, we could debug the breakpad code and figure out where it's falling down.

Comment 10

[(not currently active) Ted Mielczarek]

It's quite possible that bug 514188 will fix this.

Comment 11

[Saint Wesonga]

(In reply to comment #10)
> It's quite possible that bug 514188 will fix this.

I don't think it has (assuming that that fix is in my Firefox 3.6.4 build 6) since I just ran into this - see http://crash-stats.mozilla.com/report/index/faf28c45-38ba-416e-aa9d-2a0612100607

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #9)
> Yeah, that's a bummer. If we find a reproducible crash that triggers this,
> we could debug the breakpad code and figure out where it's falling down.

We can more or less reproduce this crash for our Firefox UI update tests (bug 1202375 and bug 1222197) on Windows only. This is happening together with Marionette. I'm going to add the minidump, extra file, and more information to bug 1202375.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I uploaded the minidump as attachment 8698022 [details]. When I tried to open it with windbg the tool failed with an error 0x80004005. Looks like we indeed still fail to produce valid minidumps here.

Comment 14

[(not currently active) Ted Mielczarek]

This bug is overly-broad. Specific reproducible crashes that produce bad dumps should be filed separately.