1062676 - Marionette failed to recover after a test case crashed

Status: RESOLVED WONTFIX

(Remote Protocol :: Marionette, defect)

Description

[Ting-Yu Lin [:TYLin] (PST, UTC-8)]

After b2g emulator crashed in test_touchcaret.py TouchCaretTest.test_contenteditable_move_caret_to_the_right_by_one_character (bug 1062671), it failed to recover to run those test cases followed.

See log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=46360266&tree=Try&full=1

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

We have crash recovery in gaiatest, however this suggests that we should lift this into the Marionette test runner. See https://github.com/mozilla-b2g/gaia/blob/00b0859b0d5379221a72c0f9e402d2550155c9b9/tests/python/gaia-ui-tests/gaiatest/gaia_test.py#L731

With gaiatest we have a --restart option on the command line, which will restart B2G between tests. I suspect we'd want something similar, or to make the restart only needed when a crash is detected. Requesting info from Malini in case she has any ideas about this.

Comment 2

[Malini Das [:mdas] - Away, not checking bugmail]

What we can do here is if we detect a crash, and the marionette runner started the b2g/gecko instance, then it seems reasonable that it should restart that instance before continuing the test.

When we get an IOError here https://mxr.mozilla.org/mozilla-central/source/testing/marionette/client/marionette/marionette.py#615 ( thrown from https://mxr.mozilla.org/mozilla-central/source/testing/marionette/transport/marionette_transport/transport.py#58), then we should check if either --binary or --homedir was passed, and restart the instance.

Not sure if we want to turn this behaviour on via a command line pref or not. Seems like we should just do it by default.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Speaking in #ateam on IRC about this, it seems that very few of our test harnesses continue running when a crash is detected. We currently recover in gaiatest, but perhaps even there we should consider aborting on crash. There's a possibility that recovering will only cause us to crash multiple times and therefore create a lot of results spam. I'm going to mark this as WONTFIX, but we could always reopen if this is something we definitely want to implement.