1117395 - Set RETRY on "timed out waiting for emulator to start" and "We have not been able to establish a telnet connection with the emulator"

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Phil Ringnalda (:philor)]

These two failures amount to 20% of the orange for the past week.

Comment 1

[(PTO June 19-July 7) Ryan VanderMeulen]

Can we take bug 1109932 while we're at it?

Comment 2

[(PTO June 19-July 7) Ryan VanderMeulen]

lol, bug dep reading fail

Comment 3

[(PTO June 19-July 7) Ryan VanderMeulen]

"marionette.errors.TimeoutException: TimeoutException: Connection timed out" is another frequent one we hit on B2G

Comment 4

[Phil Ringnalda (:philor)]

The way we're currently hitting it like crazy on mozilla-inbound, but not hitting it on any other tree, makes me doubt that it's a safe thing to set RETRY on. Maybe if we actually filed it, and it had a demonstrated history of being starred across multiple trees without new code in common, it would seem safer.

Comment 5

[(PTO June 19-July 7) Ryan VanderMeulen]

Good call, it was from bug 1099152.

Comment 6

[(PTO June 19-July 7) Ryan VanderMeulen]

I'm staring at a pile of about 40 of these on inbound right now. What can we do to bump the priority on this if we can't fix the underlying issue?

Comment 7

[(PTO June 19-July 7) Ryan VanderMeulen]

The referenced bugs in the deps are #1 and #2 on OrangeFactor at the moment.

Comment 8

[(PTO June 19-July 7) Ryan VanderMeulen]

Bug 1109932 is hitting frequently enough that I'm closing all trees that run tests on Android.

Comment 9

[Chris Cooper [:coop] (he/him)]

I think this is related to bug 1013634, the patch for which was backed out last week according to https://bugzilla.mozilla.org/show_bug.cgi?id=1013634#c19.

Comment 10

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #9)
> I think this is related to bug 1013634, the patch for which was backed out
> last week according to
> https://bugzilla.mozilla.org/show_bug.cgi?id=1013634#c19.

Or not.

I've backed out Armen's mozharness patch:

http://hg.mozilla.org/build/mozharness/rev/27e55b4b5c9a

Comment 11

[(PTO June 19-July 7) Ryan VanderMeulen]

Retriggered jobs are coming back green. Trees reopened.

Comment 12

[Chris Cooper [:coop] (he/him)]

I think we're out of the woods here. We'll tackled the intermittent issues in bug 1109932.

Comment 13

[(PTO June 19-July 7) Ryan VanderMeulen]

I think this bug as-filed is still valid (or WONTFIX if you don't want to go that route). Note that there are B2G failures as well that also were a motivation for this bug being filed.