874822 - [B2G] Try and recover Marionette session for following test when b2g process crashes

Status: RESOLVED FIXED

(Remote Protocol :: Marionette, defect)

Description

[Zac C (:zac)]

Attachment: test case for b2g process crash

For the last week or so the b2g process has been crashing more regularly than it was and occasionally we lose a whole test run from it.

The reason for the crashes aside (for which there is no clear pattern yet), Marionette fails to wait for the b2g process to restart and recover the session for the following test.

This might be closely related to bug 874599.

I've attached a test case for replicating it.

Comment 1

[Zac C (:zac)]

Attachment: Console log of test case run

Comment 2

[Jonathan Griffin (:jgriffin)]

This is approximately the same issue as bug 874599, but the solution needs to be slightly different.  We need to be able to invoke the code to restart B2G even before we create a new session (since that will fail if B2G has crashed).  Will have to think about how to best implement this.

Comment 3

[Malini Das [:mdas] - Away, not checking bugmail]

We should also grab crashreporter logs before restarting

Comment 4

[Zac C (:zac)]

Today there was a crash resulting in a broken pipe.
http://pastebin.mozilla.org/2561570
http://qa-selenium.mv.mozilla.com:8080/job/b2g.unagi.mozril.gaia.master.ui/31/ (volatile link)

The unusual thing about this crash is that it did not attempt to run the remainder of the tests and it did not provide any summary of pass/fail/total as it used to.

Comment 5

[Zac C (:zac)]

We've moved up to 0.5.32 now.

Comment 6

[Jonathan Griffin (:jgriffin)]

So, we've fixed the problem where the session isn't cleaned up when B2G crashes during a test.  We now have a problem that the next test fails because it dies (due to lack of an active Marionette instance) before gaiatest reboots B2G.

Stack trace:

----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jgriffin/gaia-ui-tests/venv/local/lib/python2.7/site-packages/marionette_client-0.5.35-py2.7.egg/marionette/marionette_test.py", line 123, in run
    self.setUp()
  File "/home/jgriffin/gaia-ui-tests/gaiatest/gaia_test.py", line 416, in setUp
    MarionetteTestCase.setUp(self)
  File "/home/jgriffin/gaia-ui-tests/venv/local/lib/python2.7/site-packages/marionette_client-0.5.35-py2.7.egg/marionette/marionette_test.py", line 301, in setUp
    CommonTestCase.setUp(self)
  File "/home/jgriffin/gaia-ui-tests/venv/local/lib/python2.7/site-packages/marionette_client-0.5.35-py2.7.egg/marionette/marionette_test.py", line 236, in setUp
    self.marionette.start_session()
  File "/home/jgriffin/gaia-ui-tests/venv/local/lib/python2.7/site-packages/marionette_client-0.5.35-py2.7.egg/marionette/marionette.py", line 446, in start_session
    self.session = self._send_message('newSession', 'value')
  File "/home/jgriffin/gaia-ui-tests/venv/local/lib/python2.7/site-packages/marionette_client-0.5.35-py2.7.egg/marionette/marionette.py", line 337, in _send_message
    response = self.client.send(message)
  File "/home/jgriffin/gaia-ui-tests/venv/local/lib/python2.7/site-packages/marionette_client-0.5.35-py2.7.egg/marionette/client.py", line 85, in send
    self.connect()
  File "/home/jgriffin/gaia-ui-tests/venv/local/lib/python2.7/site-packages/marionette_client-0.5.35-py2.7.egg/marionette/client.py", line 73, in connect
    hello = self.receive()
  File "/home/jgriffin/gaia-ui-tests/venv/local/lib/python2.7/site-packages/marionette_client-0.5.35-py2.7.egg/marionette/client.py", line 58, in receive
    status=ErrorCodes.INVALID_RESPONSE)
TEST-UNEXPECTED-FAIL | test_crash.py TestB2GProcessCrash.test_b_second | InvalidResponseException: Could not successfully complete transport of message to Gecko, socket closed?

This is tricky to fix.  It will certainly require changes to gaiatest, and perhaps to Marionette as well.

Comment 7

[Jonathan Griffin (:jgriffin)]

I think what we need to do is implement an object which we pass to MarionetteTestCase that can be used to cache values (like is_android_build) between tests, and that tests will have access to.  

This will allow us to avoid having to require a Marionette session after the first test in order to figure out if we're running on a device or a desktop build, for example, and will let us handle the reboot procedure properly.

Comment 8

[Jonathan Griffin (:jgriffin)]

Proposed a gaiatest-only fix (no Marionette changes needed) in https://github.com/mozilla/gaia-ui-tests/pull/1125.

Comment 9

[Jonathan Griffin (:jgriffin)]

https://github.com/mozilla/gaia-ui-tests/commit/98973e7ad8695cb339f3162a690e8c9b2dfcd6e4