Closed Bug 1154804 Opened 9 years ago Closed 9 years ago

Android 4.3 Debug mochitest jobs and individual tests run for too long

Categories

(Testing :: General, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(firefox40 affected)

RESOLVED FIXED
Tracking Status
firefox40 --- affected

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Once a few assertions are avoided, the biggest problem with Android 4.3 Debug mochitests is that they run slowly -- significantly more slowly than the corresponding tests on Android 4.3 Opt, or on Android 4.0 Debug (pandas).

Individual tests run more slowly, and the longer tests exceed the 330 second mochitest time-out. At least 40 individual mochitests time-out this way frequently; more fail infrequently. If I increase the default mochitest timeout to 930 seconds, almost all of these timeouts are reliably avoided. (I have not tried any timeout between 330 and 930 seconds.)

An additional problem is that the test jobs run for too long, exceeding the 60 minute "application" timeout. At 16 chunks, several mochitest chunks take about 140 minutes to complete on Android 4.3 Debug.
I think there are 3 ways forward:

1. Increase the mochitest per-test timeout and the mochitest job timeout on Android 4.3 Debug. That will produce a green run, but I think the sheriffs will object because it will increase the time it takes to validate a push (most jobs on other platforms will complete in 30 to 60 minutes, but you would need to wait another 60+ minutes to see if Android 4.3 Debug was regressed).

2. Increase the number of chunks and either increase the mochitest per-test timeout or skip those 40-80 long-running tests. I expect we would need about 40 chunks for plain mochitests.

3. Use a bigger, faster aws instance type for mochitests. I think this is the best solution for the tests, but likely increases our aws costs. I don't know what instance type is needed.
Ouch, that's painfully slow.  My experience with B2G emulator tests was that we needed a significantly faster VM to produce any meaningful difference in runtimes.  I agree this is probably the best option, but will likely take releng some time to roll out.  In the meantime, we may want to skip the slow-running tests.
:kmoir -- What are your thoughts here, especially on faster aws vs mega-chunks?
Flags: needinfo?(kmoir)
Mochitests currently run on m1.medium for Android 4.3 debug.  As a first step, we could try running them on c3.xlarge like reftests and friends.
Flags: needinfo?(kmoir)
Depends on: 1160183
With bug 1160183 Android 4.3 debug mochitests run on c3.xlarge. That reduced test job times to between 30 and 60 minutes -- very similar to Android 4.3 opt mochitests (on m1.medium). Also, nearly all individual test timeouts were eliminated by the change. Perfect!
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Assignee: nobody → gbrown
Blocks: 1164996
No longer blocks: 1164996
See Also: → 1164996
You need to log in before you can comment on or make changes to this bug.