If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Timeouts running jit-tests on Android cause infinite retries

RESOLVED FIXED in mozilla37

Status

Testing
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: dminor, Assigned: dminor)

Tracking

(Blocks: 1 bug)

Trunk
mozilla37
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

3 years ago
It looks like the way we handle timeouts in the jit-test.py script causes infinite retries in automation [1].

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1098508#c4
(Assignee)

Updated

3 years ago
Summary: Timeouts running jit-tests on Android cause infinte retries → Timeouts running jit-tests on Android cause infinite retries
(Assignee)

Comment 1

3 years ago
gbrown located the magic in question:
http://mxr.mozilla.org/build/source/buildbotcustom/status/errors.py#12
Assignee: nobody → dminor
Status: NEW → ASSIGNED
(Assignee)

Comment 2

3 years ago
I've been trying two approaches here:

The first is to handle the exception right at the point where the individual tests are run e.g. [1]. It looks like if we hit one of these adb device timeouts, the remainder of the tests will timeout and the test run will be marked busted (red) rather than failed (orange).

The second is to handle the exception in the generator that calls into the code above. This lets me end the test run prematurely e.g. [2] and [3] and seems like a better approach to me. The problem here is that either I'm missing something or I haven't had any luck getting one of adb device timeouts with this code despite many retriggers, so I'm not sure this works as intended.

Terrence, any preference as to how I proceed here? Thanks!

[1] https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=2d641f48db82
[2] https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=18ccf7cd4a87
[3] https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=9083c5fe9ac6
Flags: needinfo?(terrence)
I agree that handling the error as in [2] is the better approach. I guess an ADB failure is an infra problem -- I thought those were supposed to show up as red?
Flags: needinfo?(terrence)
(Assignee)

Comment 4

3 years ago
Created attachment 8544604 [details] [diff] [review]
Properly handle devicemanager exceptions

Sorry for the long delay in posting this patch for review, I haven't been able to get the timeouts to reproduce on try since November. Rather than delay further I thought it best to get it reviewed and landed. I think it is better than what we have now.

Recent try run here:
https://tbpl.mozilla.org/?tree=Try&rev=c17a2cdcd382&showall=1
Attachment #8544604 - Flags: review?(terrence)
Comment on attachment 8544604 [details] [diff] [review]
Properly handle devicemanager exceptions

Review of attachment 8544604 [details] [diff] [review]:
-----------------------------------------------------------------

Great! I expect the sheriffs will be able to tell us in a couple weeks if this worked or not.
Attachment #8544604 - Flags: review?(terrence) → review+
(Assignee)

Comment 6

3 years ago
Thanks, pushed to: https://hg.mozilla.org/integration/mozilla-inbound/rev/93640018725d
https://hg.mozilla.org/mozilla-central/rev/93640018725d
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla37
You need to log in before you can comment on or make changes to this bug.