855681 - Try to get stacks from child processes that were alive after shutdown

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Ed Morley [:emorley]]

To help debug bug 603147.

Comment 1

[Ed Morley [:emorley]]

Attachment: Part 1: Get stacks

* Uses killAndGetStack() rather than killPid()
* Passes in some now-required parameters to checkForZombies()
* Tweaks parameters for killAndGetStack(), since we only have the process ID in checkForZombies(), not the process object.

Comment 2

[Ed Morley [:emorley]]

Attachment: Part 2: Pyflakes

Comment 3

[Ed Morley [:emorley]]

https://tbpl.mozilla.org/?tree=Try&rev=c9d5ccae9936

However even after lots of retriggers, I still hadn't hit bug 603147, so no idea if the patches here actually work, or if they're actually a pile of nonsense. Same ol', same ol' :-)

Comment 4

[(not currently active) Ted Mielczarek]

Comment on attachment 730645 [details] [diff] [review]
Part 1: Get stacks

Review of attachment 730645 [details] [diff] [review]:
-----------------------------------------------------------------

::: build/automation.py.in
@@ +1024,5 @@
>          return
>        elif self.IS_WIN32:
>          # We should have a "crashinject" program in our utility path
>          crashinject = os.path.normpath(os.path.join(utilityPath, "crashinject.exe"))
>          if os.path.exists(crashinject) and subprocess.Popen([crashinject, str(proc.pid)]).wait() == 0:

This needs to be changed to processPID as well.

Comment 5

[(not currently active) Ted Mielczarek]

Comment on attachment 730646 [details] [diff] [review]
Part 2: Pyflakes

Review of attachment 730646 [details] [diff] [review]:
-----------------------------------------------------------------

Feel free to just fold these patches together.

::: build/automation.py.in
@@ +1024,5 @@
>          return
>        elif self.IS_WIN32:
>          # We should have a "crashinject" program in our utility path
>          crashinject = os.path.normpath(os.path.join(utilityPath, "crashinject.exe"))
> +        if os.path.exists(crashinject) and subprocess.Popen([crashinject, str(processPID)]).wait() == 0:

This change should really be in the previous patch.

Comment 6

[Ed Morley [:emorley]]

Thank you for the review (yeah sorry that change was meant to be in the first patch); landed folded:
https://hg.mozilla.org/integration/mozilla-inbound/rev/1c3872e0e292

Comment 7

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/1c3872e0e292

Comment 8

[Ed Morley [:emorley]]

Hmm:

00:58:54     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 1732
00:58:54     INFO -  INFO | zombiecheck | Checking for orphan process with PID: 1068
00:58:54  WARNING -  TEST-UNEXPECTED-FAIL | zombiecheck | child process 1068 still alive after shutdown
00:58:54     INFO -  Error 5 opening target process
00:58:54     INFO -  Can't trigger Breakpad, just killing process

Comment 9

[(not currently active) Ted Mielczarek]

What OS was that?

Comment 10

[Ed Morley [:emorley]]

Win8:
https://tbpl.mozilla.org/php/getParsedLog.php?id=21462224&tree=Mozilla-Inbound#error0

Comment 11

[(not currently active) Ted Mielczarek]

Maybe crashinject just doesn't work on Win8?

Comment 12

[Ed Morley [:emorley]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #11)
> Maybe crashinject just doesn't work on Win8?

We're also seeing this on Win7 :-(

https://tbpl.mozilla.org/php/getParsedLog.php?id=21631665&tree=Mozilla-Inbound#error0

Comment 13

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Any progress here?

Comment 14

[(not currently active) Ted Mielczarek]

I have another approach to getting minidumps I'm testing for bug 874647, simply calling MiniDumpWriteDump directly using ctypes:
https://tbpl.mozilla.org/?tree=Try&rev=2dce007c810a

We might be able to try that instead.

Comment 15

[Ed Morley [:emorley]]

The patches here landed, and other bugs are covering anything left -> closing out.