Closed Bug 892688 Opened 6 years ago Closed 6 years ago

Improve unit test reliability on Android x86 emulator

Categories

(Testing :: General, defect)

x86
Android
defect
Not set

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 936226

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

(Whiteboard: [reit-x86])

Unit tests run well on the Android x86 emulators, but not perfectly. Consider these failure counts from a recent run:

M1   571
M2   109
M3     9
M4     0
M5     0
M6     0
M7     1
M8    32

RC1    4
RC2    5

R1     did not complete
R2     did not complete
R3     did not complete
R4     did not complete

C      did not complete
J1     0
J2     0
J3     1

(reftests did not complete because run-time exceeded the default 60 minute timeout).
Assignee: nobody → gbrown
Depends on: 897970
Depends on: 898036
I tried increasing the number of reftest chunks to 16, but 3 chunks still ran for more than 60 minutes and did not complete. 1 other chunk crashed. The remaining 8 chunks all had substantial errors -- about 400 failed tests across the 8 "good" chunks. Very disappointing.

I tried changing the emulator screen size and density to match the tegras and re-ran the reftests: there was a significant improvement!

R1     1
R2     2
R3    11
R4     0
R5     6
R6    11
R7     1
R8     2
R9     1
R10    0
R11    0
R12    0
R13    0
R14    0
R15    1
R16    5

(This is unexpected, since these tests are running with --ignore-window-size...perhaps something to investigate in future.)
Depends on: 899605
Depends on: 899614
Depends on: 899655
Depends on: 899794
Depends on: 900664
xpcshell tests run well: 253 passed / 4 failed.
Depends on: 902069
Depends on: 902073
Depends on: 902081
Depends on: 902089
Depends on: 902160
I found that the --certificate-path was not being set correctly for x86, and tweaked a couple of other x86 mozharness parameters. With those latest changes, mochitests behave very much the same as on tegra/panda. Using android.json (instead of androidx86.json), only 4 tests failed. With the androidx86.json update in bug 902160, everything passes:

M1   571  -> 0
M2   109     0
M3     9     0
M4     0     0
M5     0     0
M6     0     0
M7     1     0
M8    32     0
Whiteboard: [reit-x86]
Depends on: 907383
No longer depends on: 902073
No longer depends on: 899605
No longer depends on: 900664
Depends on: 912519
Depends on: 913627
Depends on: 917053
Depends on: 917508
Depends on: 917543
Depends on: 919784
No longer depends on: 917508
It seems that making the emulator be more stable is part of the requirements of making the jobs visible on tbpl.
Blocks: 917361
No longer blocks: 891959
Depends on: 917562
Depends on: 927476
Depends on: 927602
gbrown, would you please give a status update of the dep bugs to this one?
I want to understand the big picture and see where we are.

Thanks!
Dep bugs copied to bug 936226, which is really the same bug.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 936226
You need to log in before you can comment on or make changes to this bug.