1127082 - Frequent intermittent failures in xpcshell tests on Android 4.4 emulator

Status: RESOLVED DUPLICATE of bug 1127928

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

When running xpcshell tests on the experimental Android 4.4 emulator, I see intermittent failures that seem to be related to missing test files.

For instance, http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/[e-mail]-812c76716cc4/try-android-api-9/try_ubuntu64_vm_mobile_test-xpcshell-2-bm118-tests1-linux64-build628.txt.gz:

18:54:25     INFO -  TEST-PASS | parser/xml/test/unit/test_xml_declaration.js | took 5237ms
18:54:25     INFO -  TEST-START | parser/xml/test/unit/test_sanitizer.js
18:54:25  WARNING -  TEST-UNEXPECTED-FAIL | parser/xml/test/unit/test_sanitizer.js | xpcshell return code: 0
18:54:25     INFO -  TEST-INFO took 4486ms
18:54:25     INFO -  >>>>>>>
18:54:25     INFO -  PROCESS | parser/xml/test/unit/test_sanitizer.js | xpcw: cd /sdcard/tests/xpcshell/parser/xml/test/unit
18:54:25     INFO -  PROCESS | parser/xml/test/unit/test_sanitizer.js | xpcw: xpcshell -r /sdcard/tests/xpcshell/c/httpd.manifest --greomni /data/local/xpcb/fennec-38.0a1.en-US.android-arm.apk -m -s -e const _HTTPD_JS_PATH = "/sdcard/tests/xpcshell/c/httpd.js"; -e const _HEAD_JS_PATH = "/sdcard/tests/xpcshell/head.js"; -e const _TESTING_MODULES_DIR = "/sdcard/tests/xpcshell/m"; -f /sdcard/tests/xpcshell/head.js -e const _SERVER_ADDR = "localhost" -e const _HEAD_FILES = []; -e const _TAIL_FILES = []; -e const _JSDEBUGGER_PORT = 0; -e const _TEST_FILE = ["test_sanitizer.js"]; -e const _TEST_NAME = "parser/xml/test/unit/test_sanitizer.js" -e _execute_test(); quit(0);
18:54:25     INFO -  PROCESS | parser/xml/test/unit/test_sanitizer.js | JavaScript warning: /sdcard/tests/xpcshell/head.js, line 54: JavaScript 1.7's let blocks are deprecated
18:54:25     INFO -  PROCESS | parser/xml/test/unit/test_sanitizer.js | JavaScript warning: /sdcard/tests/xpcshell/head.js, line 74: JavaScript 1.7's let blocks are deprecated
18:54:25     INFO -  PROCESS | parser/xml/test/unit/test_sanitizer.js | JavaScript warning: /sdcard/tests/xpcshell/head.js, line 106: JavaScript 1.7's let blocks are deprecated
18:54:25     INFO -  (xpcshell/head.js) | test MAIN run_test pending (1)
18:54:25     INFO -  Error: cannot open file 'results.js' for reading at test_sanitizer.js:6
18:54:25     INFO -  run_test@test_sanitizer.js:6:3
18:54:25     INFO -  _execute_test@/sdcard/tests/xpcshell/head.js:498:5
18:54:25     INFO -  @-e:1:1
18:54:25     INFO -  
18:54:25     INFO -  <<<<<<<

18:54:25     INFO -  TEST-PASS | security/manager/ssl/tests/unit/test_nsCertType.js | took 24572ms
18:54:25     INFO -  Following exceptions were raised:
18:54:25     INFO -  Traceback (most recent call last):
18:54:25     INFO -    File "/builds/slave/test/build/tests/xpcshell/runxpcshelltests.py", line 159, in run
18:54:25     INFO -      self.run_test()
18:54:25     INFO -    File "/builds/slave/test/build/tests/xpcshell/runxpcshelltests.py", line 597, in run_test
18:54:25     INFO -      head_files, tail_files = self.getHeadAndTailFiles(self.test_object)
18:54:25     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 99, in getHeadAndTailFiles
18:54:25     INFO -      return (list(sanitize_list(test['head'], 'head')),
18:54:25     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 93, in sanitize_list
18:54:25     INFO -      path))
18:54:25     INFO -  Exception: head file does not exist: /sdcard/tests/xpcshell/services/common/tests/unit/head_global.js
18:54:25     INFO -  
18:54:25     INFO -  Traceback (most recent call last):
18:54:25     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 628, in <module>
18:54:25     INFO -      main()
18:54:25     INFO -    File "/builds/slave/test/build/tests/xpcshell/remotexpcshelltests.py", line 623, in main
18:54:25     INFO -      **options.__dict__):
18:54:25     INFO -    File "/builds/slave/test/build/tests/xpcshell/runxpcshelltests.py", line 1363, in runTests
18:54:25     INFO -      raise exceptions[0]
18:54:25     INFO -  Exception: head file does not exist: /sdcard/tests/xpcshell/services/common/tests/unit/head_global.js

Comment 1

[Geoff Brown [:gbrown]]

Running locally, I see even stranger behavior:
 - no files pushed to /sdcard
 - test files pushed to /data/local
 - not enough space on /data/local for test files + binaries.

Comment 2

[Geoff Brown [:gbrown]]

$ adb shell mkdir /mnt/sdcard/tests
mkdir failed for /mnt/sdcard/tests, Read-only file system

oops!

Comment 3

[Geoff Brown [:gbrown]]

hardware-qemu.ini:hw.sdCard = no

s/no/yes gives us a writable /mnt/sdcard with 500M of space available and devicemanagerADB uses that for the test root.

In local runs, I still fail to push xpcshell libs to /data/local. /data on a clean emulator image has about 100M free, but that's not enough for my apk+binaries+libxul.

Comment 4

[Geoff Brown [:gbrown]]

In bug 1062365, I will generate a new avd with hw.sdCard = yes and a 600M user data partition, which gives:

$ adb shell df
Filesystem               Size     Used     Free   Blksize
/dev                   373.7M   132.0K   373.6M   4096
/sys/fs/cgroup         373.7M    12.0K   373.7M   4096
/mnt/asec              373.7M     0.0K   373.7M   4096
/mnt/obb               373.7M     0.0K   373.7M   4096
/system                541.9M   260.8M   281.1M   4096
/data                  197.0M    31.5M   165.5M   4096
/cache                  65.0M     4.1M    60.9M   4096
/mnt/media_rw/sdcard   498.0M    22.0K   498.0M   2048
/storage/sdcard        498.0M    22.0K   498.0M   2048

That appears to give us enough space on /data/local and /sdcard for xpcshell tests, whether running locally or in integration (which is less because optimized libs are retrieved from the apk).

Comment 5

[Geoff Brown [:gbrown]]

After updating the 4.4 AVD with sdcard enabled and an increased /data partition size, xpcshell failures persist. Some failures are simple test failures which can likely be addressed by manifest changes -- I'll worry about those later. But other failures indicate that test files are missing, intermittently -- different files are missing on each test run.

It turns out the ADB pushDir(tests directory) command is intermittently failing, silently - bug 1127942 - after timing out - bug 1127928. Use of zip/unzip would alleviate the problem, but that's broken too - bug 1127457 - and besides, unzip is not available on the 4.4 emulator (I may fix that once everything else is sorted).

Interestingly, xpcshell-3 is not affected by missing files. That's because xpcshell-3 runs very few tests - bug 1127917 - and those tests use files that happen to be pushed early.

Comment 6

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=573202c1fc16 reveals the timeout with extra logging.
https://treeherder.mozilla.org/#/jobs?repo=try&revision=17ee50d80793 shows that increasing the pushDir timeout eliminates missing files.

Comment 7

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1892cdbb4594 confirms that we can run xpcshell tests reliably on Android 4.4. There's a little more work to do to get this to tier 1 status, but this is good enough for now.