1093328 - mozcrash may flood test log if many minidumps found

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Geoff Brown [:gbrown]]

In bug 1091916, an Android x86 crash was introduced for certain xpcshell tests. When the crash occurs, many (over 1000) minidump files are created. See https://tbpl.mozilla.org/php/getParsedLog.php?id=51563406&tree=Mozilla-Inbound&full=1.

When this happens, mozcrash dutifully calls minidump_stackwalk for each one, flooding the test log with crash report after crash report. Eventually, we hit:

Output exceeded 52428800 bytes, remaining output has been truncated (output was 52458724 bytes)

command timed out: 14400 seconds elapsed running ['/tools/buildbot/bin/python', 'scripts/scripts/android_emulator_unittest.py', '--cfg', 'android/androidx86.py', '--test-suite', 'robocop-1', '--test-suite', 'robocop-2', '--test-suite', 'robocop-3', '--test-suite', 'xpcshell', '--blob-upload-branch', 'mozilla-inbound', '--download-symbols', 'ondemand'], attempting to kill
process killed by signal 9

It is not clear to me if the time-out is  caused by all the time spent dumping crash reports, log processing, or something else.

And I don't understand why we are generating so many minidump files.

But I think we can handle this situation better: If more than N minidump files are found by mozcrash, let's just process and report the first N. N = 10, maybe?

Comment 1

[(not currently active) Ted Mielczarek]

That makes sense. This is a pretty broken situation to get into.

Comment 2

[Geoff Brown [:gbrown]]

Attachment: do not report more than 10 crash reports at a time

Is 10 a good number?

Could/should we limit this in breadpad also, or in addition?

https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=7e4fe39a7ac7

Comment 3

[(not currently active) Ted Mielczarek]

Comment on attachment 8516705 [details] [diff] [review]
do not report more than 10 crash reports at a time

Review of attachment 8516705 [details] [diff] [review]:
-----------------------------------------------------------------

Seems reasonable for an arbitrary limit. Might make that a .warn or something more serious.

Comment 4

[Geoff Brown [:gbrown]]

Changed to .warning - thanks.

https://hg.mozilla.org/integration/mozilla-inbound/rev/e80e4fa3a93d

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/e80e4fa3a93d