1329528 - OSError: [Errno 1] Operation not permitted exception when killing a zombie process.

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Glenn Watson [:gw]]

On OSX, calling the kill() method on the Process class can result in an unhandled OSError exception.

This occurs when the process in question has exited and is in the zombie state.

An example stack trace from [1] is:

Tests with unexpected results:
  ▶ ERROR [expected OK] /html/browsers/windows/noreferrer.html
  └   → Traceback (most recent call last):
  File "/Users/servo/buildbot/slave/mac-rel-wpt1/build/tests/wpt/harness/wptrunner/executors/base.py", line 149, in run_test
    result = self.do_test(test)
  File "/Users/servo/buildbot/slave/mac-rel-wpt1/build/tests/wpt/harness/wptrunner/executors/executorservo.py", line 143, in do_test
    self.proc.kill()
  File "/Users/servo/buildbot/slave/mac-rel-wpt1/build/python/_virtualenv/lib/python2.7/site-packages/mozprocess/processhandler.py", line 766, in kill
    self.proc.kill(sig=sig)
  File "/Users/servo/buildbot/slave/mac-rel-wpt1/build/python/_virtualenv/lib/python2.7/site-packages/mozprocess/processhandler.py", line 172, in kill
    send_sig(signal.SIGTERM)
  File "/Users/servo/buildbot/slave/mac-rel-wpt1/build/python/_virtualenv/lib/python2.7/site-packages/mozprocess/processhandler.py", line 159, in send_sig
    os.killpg(pid, sig)
OSError: [Errno 1] Operation not permitted

When a process (group) is in zombie state, it will remain that way until waitpid() is called on it. I tested catching the exception when os.killpg() is called, and then calling os.waitpid(pid). This appears to fix the problem locally for my test case, but I'm not sure if this is the correct fix. 

[1] https://github.com/servo/servo/pull/14818

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This could be the reason for various test failures we see in Marionette restart tests.

Comment 2

[sgiles]

There's a reproducable test case here (including the fix): https://github.com/servo/servo/pull/15579#issuecomment-280215792

Will add a patch for mozprocess tomorrow.

Comment 3

[sgiles]

Attachment: Bug 1329528 - Reap zombie processes on Mac OS if killing the process group initially fails with EPERM;

Review commit: https://reviewboard.mozilla.org/r/112940/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/112940/

Comment 4

[sgiles]

It would've been great to get a test case for this, but Python multiprocessing is too good at making sure you don't end up with Zombies.. :P

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8837939 [details]
Bug 1329528 - Reap zombie processes on Mac OS if killing the process group initially fails with EPERM;

https://reviewboard.mozilla.org/r/112940/#review114466

::: testing/mozbase/mozprocess/mozprocess/processhandler.py:169
(Diff revision 1)
> +                            # before continuing
> +                            # Note: A negative pid refers to the entire process
> +                            # group
> +                            if retries < 1 and getattr(e, "errno", None) == errno.EPERM:
> +                                try:
> +                                    os.waitpid(-pid, 0)

What will happen if we do this call by default before sending killpg()? Would that cause a freeze?

I just wonder if we could get rid of all the extra code the patch will add.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to sgiles from comment #4)
> It would've been great to get a test case for this, but Python
> multiprocessing is too good at making sure you don't end up with Zombies.. :P

We have at least a forking proc written in C in the tests folder of mozprocess. Maybe you could add just another one?

Comment 7

[Andrew Halberstadt [:ahal]]

Comment on attachment 8837939 [details]
Bug 1329528 - Reap zombie processes on Mac OS if killing the process group initially fails with EPERM;

https://reviewboard.mozilla.org/r/112940/#review114520

Thanks for digging into this! Looks reasonable to me

Comment 8

[sgiles]

Comment on attachment 8837939 [details]
Bug 1329528 - Reap zombie processes on Mac OS if killing the process group initially fails with EPERM;

https://reviewboard.mozilla.org/r/112940/#review114466

> What will happen if we do this call by default before sending killpg()? Would that cause a freeze?
> 
> I just wonder if we could get rid of all the extra code the patch will add.

Yep, we need to send the kill signal first, otherwise we'll end up waiting on non-zombie processes and potentially hanging.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2b4830caedd4
Reap zombie processes on Mac OS if killing the process group initially fails with EPERM; r=ahal

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/2b4830caedd4

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This issue also affects branches down to current beta (52.0). We might want to wait some days but then request an uplift of the patch. It would be great to see it also fixed in the next ESR release.

Comment 12

[Ryan VanderMeulen [:RyanVM][PTO June 24-28]]

No known fallout I'm aware of so far. Going to give this a go on Aurora for a bit before deciding on Beta.
https://hg.mozilla.org/releases/mozilla-aurora/rev/63baf28e129e

Comment 13

[Ryan VanderMeulen [:RyanVM][PTO June 24-28]]

Bah, this needs to be rebased around bug 1309060 if we want to uplift this to 52.

Comment 14

[Ryan VanderMeulen [:RyanVM][PTO June 24-28]]

Would still consider a rebased patch for ESR52 if it's practical to do so, but it's too late for Fx52 at this point.