2052432 - Upload a minidump for force-killed hung xpcshell processes on timeout

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, task)

Description

[Florian Quèze [:florian]]

When the xpcshell harness force-kills a hung test process on timeout, the minidump it takes is currently discarded: testTimeout returns before run_test's normal checkForCrashes, so the dump is never saved to the upload directory (and on Linux/macOS postCheck's SIGKILL races the in-progress SIGABRT Breakpad dump, truncating it). A hung/timed-out test therefore leaves no dump to inspect.

Make the harness preserve it: after killTimeout, wait (bounded) for the process to finish writing its minidump, then run checkForCrashes, which copies the .dmp to the upload dir as a test artifact and symbolicates its thread stacks into the log. The test is still reported as TIMEOUT, and the log states the dump is the force-killed process, not a real crash (buffered as a test message so a retried run keeps it out of the failure summary).

This matters most for a hard deadlock in the profiler itself (bug 1687510): the sampler thread is stuck, so no profile can be produced, and the force-killed process's minidump is the only diagnostic.

Depends on bug 2052043 (builds on its reportTimeoutResult / self-exit / safety-net-kill work). With both bugs fixed we should now have profiles uploaded for timeouts due to the test taking too long, and we should have a crash dump uploaded for hard deadlocks (especially profiler deadlocks).

Comment 1

[Florian Quèze [:florian]]

Attachment: artificial-profiler-deadlock.patch

I used this patch to reproduce a(n artificial) profiler deadlock on try. For it to cause a deadlock, the try push needs to be made with --env MOZ_TEST_PROFILER_DEADLOCK=1.

Here's the try result of this bug: https://treeherder.mozilla.org/jobs?repo=try&revision=25b739100566d72889ed1445514c316f6311b6d2

Comment 2

[Florian Quèze [:florian]]

Attachment: Bug 2052432 - Upload a minidump for force-killed hung xpcshell processes on timeout, r=ahal.

When the harness force-kills a hung test on timeout, testTimeout returned before
run_test's normal checkForCrashes, so the minidump was never saved to the upload
directory. On Linux/macOS kill_and_get_minidump only sends SIGABRT (the process
writes the dump itself via Breakpad), and postCheck's SIGKILL raced the
in-progress dump.

Wait, bounded by TIMEOUT_MINIDUMP_WAIT, for the process to finish writing its
minidump, then run checkForCrashes so the dump is uploaded as a test artifact and
its thread stacks are symbolicated into the log. The test is still reported as
TIMEOUT; the "not a real crash" note is buffered as a test message so a retried
run keeps it out of the failure summary.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/7a8d0d9f7a35
https://hg.mozilla.org/integration/autoland/rev/c872f1cf24e1
Upload a minidump for force-killed hung xpcshell processes on timeout, r=ahal.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/f5751ad9a39e
https://hg.mozilla.org/integration/autoland/rev/67c45fd4ba0a
Revert "Bug 2052432 - Upload a minidump for force-killed hung xpcshell processes on timeout, r=ahal." for causing Xpcshell failures.

Comment 5

[Atila Butkovits]

Backed out for causing Xpcshell failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/67c45fd4ba0a

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=windows%2C11%2C25h2%2Copt%2Cxpcshell%2Ctests%2Ctest-windows11-64-25h2%2Fopt-xpcshell%2Cx&revision=c872f1cf24e1ab6719fc231b7c6913a1f75ebe96&selectedTaskRun=a5LF0uA6R1y8w3Esa4krCg.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=577652790&repo=autoland&task=a5LF0uA6R1y8w3Esa4krCg.0&lineNumber=25638

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/9c91ab8a575e
https://hg.mozilla.org/integration/autoland/rev/c36edac9aec7
Upload a minidump for force-killed hung xpcshell processes on timeout, r=ahal.

Comment 7

[amarc]

https://hg.mozilla.org/mozilla-central/rev/c36edac9aec7