1745663 - fix StackwalkerErrorSignatureRule to add mdsw status with rust-minidump minidump-stackwalk

Status: RESOLVED FIXED

(Socorro :: Signature, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

The StackwalkerErrorSignatureRule uses the mdsw_status_string for signature generation.

mdsw_status_string is derived from json_dump.status in the processed crash. That's always "OK".

https://github.com/luser/rust-minidump/blob/master/minidump-processor/json-schema.md

Given that, we should remove this rule from signature generation and related documentation.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Oops--this turns out to be a difference between rust-minidump minidump-stackwalk and breakpad minidump_stackwalk. The latter does produce status, but the former always sets it to OK.

Here's some signatures that change if we remove the StackwalkerErrorSignatureRule:

app@socorro:/app$ socorro-cmd fetch_crashids --num=100 --signature=EMPTY | socorro-cmd signature --different-only
Crash id: c4d5382c-9b86-4e89-bb0b-8bf170211211
Original: OOM | large | EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER
New:      OOM | large | EMPTY: no crashing thread identified
Same?:    False
Notes:    (1)
          SignatureGenerationRule: CSignatureTool: no crashing thread identified

Crash id: 12559dd1-d7c8-497b-b1ca-497d10211211
Original: OOM | large | EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER
New:      OOM | large | EMPTY: no crashing thread identified
Same?:    False
Notes:    (1)
          SignatureGenerationRule: CSignatureTool: no crashing thread identified

Crash id: 16ab80ed-26a9-47f9-aab0-515cb0211211
Original: OOM | large | EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER
New:      OOM | large | EMPTY: no crashing thread identified
Same?:    False
Notes:    (1)
          SignatureGenerationRule: CSignatureTool: no crashing thread identified

Crash id: 934fb7f4-2690-496d-b95e-91c930211211
Original: EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER
New:      EMPTY: no crashing thread identified
Same?:    False
Notes:    (1)
          SignatureGenerationRule: CSignatureTool: no crashing thread identified

Crash id: 077d263f-c74a-41cb-8e3c-71a8a0211211
Original: EMPTY: no crashing thread identified; ERROR_NO_THREAD_LIST
New:      EMPTY: no crashing thread identified
Same?:    False
Notes:    (1)
          SignatureGenerationRule: CSignatureTool: no crashing thread identified

Crash id: 2d89a0c5-80ea-49a3-8822-b7f780211211
Original: EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER
New:      EMPTY: no crashing thread identified
Same?:    False
Notes:    (1)
          SignatureGenerationRule: CSignatureTool: no crashing thread identified

I'll need to find that information elsewhere in the json_dump and support it in the signature instead of removing this rule.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

There are three status codes we see:

ERROR_NO_MINIDUMP_HEADER

This shows up predominantly (at least as far as I can tell) when the minidump is 0-bytes. I added a shortcut to the processor to detect this situation and skip running minidump-stackwalk, so I'll fix the mdsw_status_string for that.

ERROR_NO_THREAD_LIST

minidump-stackwalk emits "[ERROR] Error processing dump: ERROR_NO_THREAD_LIST: The thread list stream was not found" to stderr. I can fix the processor rule to change mdsw_status_string to "ERROR_NO_THREAD_LIST" when we see "thread list stream was not found" in stderr. I wrote up this issue to add an error code that's less fragile to stderr message:

https://github.com/luser/rust-minidump/issues/354

I'll fix the first one in a legit way and the second one in a hacky way with a FIXME now.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

Aria implemented a fix that's different (and better) than what I was thinking. Instead of error codes like ERROR_NO_THREAD_LIST, stderr will have the exception type. So then we'll end up with something like MissingThreadsList as an indicator.

That means we'll have some changes in crash signatures like variations on the following:

old	new
EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER	EMPTY: no crashing thread identified; EmptyMinidump
EMPTY: no crashing thread identified; ERROR_NO_THREAD_LIST	EMPTY: no crashing thread identified; MissingThreadList

I'm adjusting the MinidumpStackwalkerRule accordingly and we won't have to do a hacky/fragile fix after all.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5940: bug 1745663: fix error codes in crash signatures

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5940: "bug 1745663: fix error codes in crash signatures" in 3ba22b1.

I'll keep an eye on crash signatures with EMPTY in them and we'll see how they change over the next 24 hours and whether we caught everything.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 5942: bug 1745663: fix KeyError caused by CrashingThreadRule

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR #5942: "bug 1745663: fix KeyError caused by CrashingThreadRule" in 22eea03.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

This went to production 8 hours ago or so in bug #1745983. Marking as FIXED.