Closed Bug 892688 Opened 6 years ago Closed 6 years ago
Improve unit test reliability on Android x86 emulator
Unit tests run well on the Android x86 emulators, but not perfectly. Consider these failure counts from a recent run: M1 571 M2 109 M3 9 M4 0 M5 0 M6 0 M7 1 M8 32 RC1 4 RC2 5 R1 did not complete R2 did not complete R3 did not complete R4 did not complete C did not complete J1 0 J2 0 J3 1 (reftests did not complete because run-time exceeded the default 60 minute timeout).
I tried increasing the number of reftest chunks to 16, but 3 chunks still ran for more than 60 minutes and did not complete. 1 other chunk crashed. The remaining 8 chunks all had substantial errors -- about 400 failed tests across the 8 "good" chunks. Very disappointing. I tried changing the emulator screen size and density to match the tegras and re-ran the reftests: there was a significant improvement! R1 1 R2 2 R3 11 R4 0 R5 6 R6 11 R7 1 R8 2 R9 1 R10 0 R11 0 R12 0 R13 0 R14 0 R15 1 R16 5 (This is unexpected, since these tests are running with --ignore-window-size...perhaps something to investigate in future.)
xpcshell tests run well: 253 passed / 4 failed.
I found that the --certificate-path was not being set correctly for x86, and tweaked a couple of other x86 mozharness parameters. With those latest changes, mochitests behave very much the same as on tegra/panda. Using android.json (instead of androidx86.json), only 4 tests failed. With the androidx86.json update in bug 902160, everything passes: M1 571 -> 0 M2 109 0 M3 9 0 M4 0 0 M5 0 0 M6 0 0 M7 1 0 M8 32 0
It seems that making the emulator be more stable is part of the requirements of making the jobs visible on tbpl.
gbrown, would you please give a status update of the dep bugs to this one? I want to understand the big picture and see where we are. Thanks!
Dep bugs copied to bug 936226, which is really the same bug.
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 936226
You need to log in before you can comment on or make changes to this bug.