892688 - Improve unit test reliability on Android x86 emulator

Status: RESOLVED DUPLICATE of bug 936226

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

Unit tests run well on the Android x86 emulators, but not perfectly. Consider these failure counts from a recent run:

M1   571
M2   109
M3     9
M4     0
M5     0
M6     0
M7     1
M8    32

RC1    4
RC2    5

R1     did not complete
R2     did not complete
R3     did not complete
R4     did not complete

C      did not complete
J1     0
J2     0
J3     1

(reftests did not complete because run-time exceeded the default 60 minute timeout).

Comment 1

[Geoff Brown [:gbrown]]

I tried increasing the number of reftest chunks to 16, but 3 chunks still ran for more than 60 minutes and did not complete. 1 other chunk crashed. The remaining 8 chunks all had substantial errors -- about 400 failed tests across the 8 "good" chunks. Very disappointing.

I tried changing the emulator screen size and density to match the tegras and re-ran the reftests: there was a significant improvement!

R1     1
R2     2
R3    11
R4     0
R5     6
R6    11
R7     1
R8     2
R9     1
R10    0
R11    0
R12    0
R13    0
R14    0
R15    1
R16    5

(This is unexpected, since these tests are running with --ignore-window-size...perhaps something to investigate in future.)

Comment 2

[Geoff Brown [:gbrown]]

xpcshell tests run well: 253 passed / 4 failed.

Comment 3

[Geoff Brown [:gbrown]]

I found that the --certificate-path was not being set correctly for x86, and tweaked a couple of other x86 mozharness parameters. With those latest changes, mochitests behave very much the same as on tegra/panda. Using android.json (instead of androidx86.json), only 4 tests failed. With the androidx86.json update in bug 902160, everything passes:

M1   571  -> 0
M2   109     0
M3     9     0
M4     0     0
M5     0     0
M6     0     0
M7     1     0
M8    32     0

Comment 4

[Armen [:armenzg]]

It seems that making the emulator be more stable is part of the requirements of making the jobs visible on tbpl.

Comment 5

[Armen [:armenzg]]

gbrown, would you please give a status update of the dep bugs to this one?
I want to understand the big picture and see where we are.

Thanks!

Comment 6

[Geoff Brown [:gbrown]]

Dep bugs copied to bug 936226, which is really the same bug.