921509 - OS X "OSError: [Errno 3] No such process" in runtests.py::killAndGetStack()

Status: RESOLVED FIXED

(Testing :: Mochitest, defect)

Description

[Ed Morley [:emorley]]

eg:
https://tbpl.mozilla.org/php/getParsedLog.php?id=28459721&tree=Mozilla-Central
https://tbpl.mozilla.org/php/getParsedLog.php?id=28430118&tree=Mozilla-Inbound


15:10:33     INFO -  Traceback (most recent call last):
15:10:33     INFO -    File "/builds/slave/talos-slave/test/build/tests/mochitest/runtests.py", line 1041, in runTests
15:10:33     INFO -      onLaunch=onLaunch
15:10:33     INFO -    File "/builds/slave/talos-slave/test/build/tests/mochitest/runtests.py", line 935, in runApp
15:10:33     INFO -      self.handleTimeout(timeout, proc, utilityPath, debuggerInfo, browserProcessId)
15:10:33     INFO -    File "/builds/slave/talos-slave/test/build/tests/mochitest/runtests.py", line 1069, in handleTimeout
15:10:33     INFO -      self.killAndGetStack(browserProcessId, utilityPath, debuggerInfo, dump_screen=not debuggerInfo)
15:10:33     INFO -    File "/builds/slave/talos-slave/test/build/tests/mochitest/runtests.py", line 725, in killAndGetStack
15:10:33     INFO -      os.kill(processPID, signal.SIGABRT)
15:10:33     INFO -  OSError: [Errno 3] No such process
15:10:33     INFO -  Automation Error: Received unexpected exception while running application

Comment 1

[(not currently active) Ted Mielczarek]

The only thing I can think of here is that the process has died before we got to kill it, possibly just crashing slowly (on Mac this can take a long time while the OS X crashreporter generates a report). We should probably just wrap that in a try / except OSError and move on in life.

Comment 2

[Ed Morley [:emorley]]

This is happening pretty frequently on OS X.

Comment 3

[Jeff Hammel]

Attachment: simple try: except OSError

Comment 4

[Jeff Hammel]

(In reply to Jeff Hammel [:jhammel] from comment #3)
> Created attachment 811822 [details] [diff] [review]
> simple try: except OSError

Doesn't break linux locally; will push to try if desired.

Comment 5

[Ed Morley [:emorley]]

Comment on attachment 811822 [details] [diff] [review]
simple try: except OSError

Could we log.info("Can't trigger Breakpad, process no longer exists") ?

Comment 6

[(not currently active) Ted Mielczarek]

Comment on attachment 811822 [details] [diff] [review]
simple try: except OSError

Review of attachment 811822 [details] [diff] [review]:
-----------------------------------------------------------------

I have a feeling that if we land this we'll start seeing crash stacks out of these tests. My theory is that the process has already crashed, but hitting this exception means we skip the check_for_crashes call so we don't get to see the stack.

::: testing/mochitest/runtests.py
@@ +723,5 @@
>          # ABRT will get picked up by Breakpad's signal handler
> +        try:
> +          os.kill(processPID, signal.SIGABRT)
> +        except OSError:
> +          # https://bugzilla.mozilla.org/show_bug.cgi?id=921509

I'd drop the comment, this is what hg blame is for. Ed's suggestion to log.info something here (maybe even log.warn?) sounds reasonable as well. This is an odd situation to get to, it'd be good to note it.

Comment 7

[Jeff Hammel]

pushed to inbound with recommendations: https://hg.mozilla.org/integration/mozilla-inbound/rev/48f4c5347dad

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/48f4c5347dad