842628 - Treat PhoneTest as a suite of tests

Status: RESOLVED DUPLICATE of bug 1079923

(Testing Graveyard :: Autophone, defect)

Description

[Mark Côté [:mcote]]

Spun off of bug 842257.

runtestsremote, unlike s1s2 or smoketest, is actually a test suite, and doesn't benefit from some of the between-tests error handling at the PhoneWorkerSubProcess level.  It makes sense, however, to keep it as a suite instead of breaking it up, since it shares a lot of functionality.  We can fix this by treating all tests as suites, even if there is only one.

We should change PhoneTest.runjob() to be a sort of iterator.  PhoneWorkerSubProcess.do_job(), instead of calling runjob() once, would call something like run_next_test() repeatedly until, say, a StopIteration exception is raised (standard iterator end signal).  If a DMError bubbles up to the PhoneWorkerSubProcess object, it tries to recover the phone and, if successful, calls run_next_test() as usual.  One beneficial side effect here is that it makes it easier to retry a test.  As the FIXME comment in do_job()'s DMError handler says, we abort the test even if it were possible to recover the phone and retry it.  We should have a maximum number of attempts in this case, though, so that it doesn't try running the same test forever.

A secondary goal should be to store the current test and test suite (and, for that matter, queued suites) on disk so that we can restart autophone and have it (more or less) resume where it left off.

Comment 1

[Bob Clary [:bc] (inactive)]

As part of Bug 1079923 the unit test runner runtestsremote splits the unit tests into separate jobs based on the different config files listed and splits each of those into separate jobs for each chunk for that test. I believe this meets the desire for more between-tests error handling.

Mark shall we dupe this to Bug 1079923 and move on?