1098529 - Timeouts running jit-tests on Android cause infinite retries

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Dan Minor [:dminor]]

It looks like the way we handle timeouts in the jit-test.py script causes infinite retries in automation [1].

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1098508#c4

Comment 1

[Dan Minor [:dminor]]

gbrown located the magic in question:
http://mxr.mozilla.org/build/source/buildbotcustom/status/errors.py#12

Comment 2

[Dan Minor [:dminor]]

I've been trying two approaches here:

The first is to handle the exception right at the point where the individual tests are run e.g. [1]. It looks like if we hit one of these adb device timeouts, the remainder of the tests will timeout and the test run will be marked busted (red) rather than failed (orange).

The second is to handle the exception in the generator that calls into the code above. This lets me end the test run prematurely e.g. [2] and [3] and seems like a better approach to me. The problem here is that either I'm missing something or I haven't had any luck getting one of adb device timeouts with this code despite many retriggers, so I'm not sure this works as intended.

Terrence, any preference as to how I proceed here? Thanks!

[1] https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=2d641f48db82
[2] https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=18ccf7cd4a87
[3] https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=9083c5fe9ac6

Comment 3

[Terrence Cole [:terrence]]

I agree that handling the error as in [2] is the better approach. I guess an ADB failure is an infra problem -- I thought those were supposed to show up as red?

Comment 4

[Dan Minor [:dminor]]

Attachment: Properly handle devicemanager exceptions

Sorry for the long delay in posting this patch for review, I haven't been able to get the timeouts to reproduce on try since November. Rather than delay further I thought it best to get it reviewed and landed. I think it is better than what we have now.

Recent try run here:
https://tbpl.mozilla.org/?tree=Try&rev=c17a2cdcd382&showall=1

Comment 5

[Terrence Cole [:terrence]]

Comment on attachment 8544604 [details] [diff] [review]
Properly handle devicemanager exceptions

Review of attachment 8544604 [details] [diff] [review]:
-----------------------------------------------------------------

Great! I expect the sheriffs will be able to tell us in a couple weeks if this worked or not.

Comment 6

[Dan Minor [:dminor]]

Thanks, pushed to: https://hg.mozilla.org/integration/mozilla-inbound/rev/93640018725d

Comment 7

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/93640018725d