718496 - Mozmill fails a lot with "[Errno 10] No child processes" and doesn't send report

Status: RESOLVED FIXED

(Testing Graveyard :: Mozmill, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Today I have seen a lot of problems with Mozmill 1.5.7 on OS X while working on various dependencies for bug 718403. A dozen of times Firefox quit and didn't sent a report. I have only seen:

TEST-UNEXPECTED-FAIL | Disconnect Error: Application unexpectedly closed
[Errno 10] No child processes

Seems like there is a code path which let us exit early without sending a report. We should fix this to ensure that we always gets reports. It would partly block bug 718403 where we need consistent information across the different test-runs.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Clint, can you please have a look at this bug? It's important because in those cases we will miss to send the associated reports. One situation I was able to see it was quitting the browser during a testrun. Mozmill didn't send a report.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Could be related to sys.exit() here:

https://github.com/mozautomation/mozmill/blob/hotfix-1.5/mozmill/mozmill/__init__.py#L820

https://github.com/mozautomation/mozmill/blob/hotfix-1.5/mozmill/mozmill/__init__.py#L524

Comment 3

[Jeff Hammel]

The sys.exit() near L820 is after the report is sent; L524 is a possibility

Comment 4

[William Lachance (:wlach)]

Fixing component.

Comment 5

[cmtalbert]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> Clint, can you please have a look at this bug? It's important because in
> those cases we will miss to send the associated reports. One situation I was
> able to see it was quitting the browser during a testrun. Mozmill didn't
> send a report.

What are the tests you are running and what is the command line you are using to start Mozmill? I'm assuming since you encountered this during testing for the default to compatibile stuff you hit this running the latest beta builds?

I need all that information ^ before I can dig in here.  Thanks!

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Jeff Hammel [:jhammel] from comment #3)
> The sys.exit() near L820 is after the report is sent; L524 is a possibility

Not really sure about it because I have never seen that line:

print "WARNING | endRunner was never called. There must have been a failure in the framework."

I will try to find a reproducible case to allow us to hopefully fix it. I don't think it is related to process handling.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, I can reproduce it with Mozmill 1.5.8pre with every build of Firefox on OS X. Just start our tests and hit Cmd+Q while the tests are running.

Steps:
1. Install Mozmill 1.5.8pre (1.5.7 should also work)
2. Clone our Mozmill tests repository
3. Run mozmill -t tests/functional -b %path%
4. After some tests have been run press Cmd+Q 

This is the stacktrace:

Traceback (most recent call last):
  File "/Volumes/data/envs/mozmill-dev/bin/mozmill", line 8, in <module>
    load_entry_point('mozmill==1.5.7', 'console_scripts', 'mozmill')()
  File "/Volumes/data/code/tools/mozmill/mozmill/mozmill/__init__.py", line 851, in cli
    CLI().run()
  File "/Volumes/data/code/tools/mozmill/mozmill/mozmill/__init__.py", line 815, in run
    self.mozmill.stop(fatal=disconnected)
  File "/Volumes/data/code/tools/mozmill/mozmill/mozmill/__init__.py", line 533, in stop
    self.stop_runner(timeout=10, close_bridge=True, hard=fatal)
  File "/Volumes/data/code/tools/mozmill/mozmill/mozmill/__init__.py", line 510, in stop_runner
    self.runner.cleanup()
  File "/Volumes/data/code/tools/mozmill/mozrunner/mozrunner/__init__.py", line 545, in cleanup
    self.stop()
  File "/Volumes/data/code/tools/mozmill/mozrunner/mozrunner/__init__.py", line 528, in stop
    self.process_handler.kill()
  File "/Volumes/data/code/tools/mozmill/mozrunner/mozrunner/killableprocess.py", line 195, in kill
    os.wait()
OSError: [Errno 10] No child processes


So it looks like it's a fallout from Clint's patch on bug 658509.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I think in such a case we can silently ignore the raised exception because the given process doesn't exist anymore.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Attachment: Patch v1

If the application gets closed by the user, there is no need to wait here. So calling os.wait() doesn't make sense.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Landed on hotfix-1.5 as:
https://github.com/mozautomation/mozmill/commit/4c58715850bb771ba7816dba339d9957ef0ec6a0