1346083 - Running xpcshell-test or mochitest-chrome locally usually fails on clean install due to silent pushDir failure

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect, P1)

Description

[Geoff Brown [:gbrown]]

I'm noticing that, usually, maybe always, this sequence fails:

 - mach android-emulator --force-update
 - mach xpcshell-test netwerk/test/unit

It looks like the xpcshell tests are not successfully pushed.

If I break and re-run mach xpcshell-test, the push is successful and tests pass.

Comment 1

[Geoff Brown [:gbrown]]

 0:00.54 LOG: ProcessReader mozdevice DEBUG 0
 0:00.58 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell ls -1A /
 0:00.69 LOG: ProcessReader mozdevice DEBUG ls: Unknown option '-1'. Aborting.
 0:00.75 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell ls -a /storage/sdcard0/
 0:00.82 LOG: ProcessReader mozdevice DEBUG /storage/sdcard0/: No such file or directory
 0:00.82 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell ls -a /storage/sdcard1/
 0:00.89 LOG: ProcessReader mozdevice DEBUG /storage/sdcard1/: No such file or directory
 0:00.89 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell ls -a /storage/sdcard/
 0:00.96 LOG: ProcessReader mozdevice DEBUG .android_secure
 0:00.96 LOG: ProcessReader mozdevice DEBUG Alarms
 0:00.96 LOG: ProcessReader mozdevice DEBUG DCIM
 0:00.96 LOG: ProcessReader mozdevice DEBUG Download
 0:00.96 LOG: ProcessReader mozdevice DEBUG LOST.DIR
 0:00.96 LOG: ProcessReader mozdevice DEBUG Movies
 0:00.96 LOG: ProcessReader mozdevice DEBUG Music
 0:00.96 LOG: ProcessReader mozdevice DEBUG Notifications
 0:00.96 LOG: ProcessReader mozdevice DEBUG Pictures
 0:00.96 LOG: ProcessReader mozdevice DEBUG Podcasts
 0:00.96 LOG: ProcessReader mozdevice DEBUG Ringtones
 0:01.00 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell mkdir /storage/sdcard/tests
pushing /home/gbrown/objdirs/droid/_tests/xpcshell
 0:01.13 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell ls -a /storage/sdcard/tests/xpc/
 0:01.23 LOG: ProcessReader mozdevice DEBUG /storage/sdcard/tests/xpc/: No such file or directory
 0:01.25 LOG: MainThread mozdevice DEBUG _checkCmd - command: adb push /home/gbrown/objdirs/droid/_tests/xpcshell /storage/sdcard/tests/xpc
 0:01.28 LOG: ProcessReader mozdevice DEBUG [  0%] /storage/sdcard/tests/xpc/xpcshell.ini: 100%
 0:01.28 LOG: ProcessReader mozdevice DEBUG [  0%] /storage/sdcard/tests/xpc/docshell/test/unit_ipc/test_pb_notification_ipc.js: 100%
 0:01.28 LOG: ProcessReader mozdevice DEBUG adb: error: failed to copy '/home/gbrown/objdirs/droid/_tests/xpcshell/docshell/test/unit_ipc/test_pb_notification_ipc.js' to '/storage/sdcard/tests/xpc/docshell/test/unit_ipc/test_pb_notification_ipc.js': Operation not permitted
 0:01.31 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell ls -a /data/local/xpcb/
 0:01.38 LOG: ProcessReader mozdevice DEBUG /data/local/xpcb/: No such file or directory
 0:01.43 LOG: MainThread mozdevice DEBUG shell - command: adb shell mkdir /data/local/xpcb; echo $?
 0:01.51 LOG: ProcessReader mozdevice DEBUG 0
 0:01.54 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell ls -a /storage/sdcard/tests/xpc/tmp/
 0:01.62 LOG: ProcessReader mozdevice DEBUG /storage/sdcard/tests/xpc/tmp/: No such file or directory
 0:01.66 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell mkdir /storage/sdcard/tests/xpc/tmp
 0:01.78 LOG: MainThread mozdevice DEBUG _runCmd - command: adb shell ls -a /data/local/xpcb/defaults/pref/

Comment 2

[Geoff Brown [:gbrown]]

I see the same behavior on the Android 4.2/x86 and Android 6.0/arm emulators...all with adb 1.0.36.

Comment 3

[Geoff Brown [:gbrown]]

I see a similar problem with mach mochitest mobile/android/tests/browser/chrome -- push dir of chrome tests seems to fail silently.

Comment 4

[Geoff Brown [:gbrown]]

Notice in https://hg.mozilla.org/mozilla-central/rev/92f215ed4df3 (bug 1328301), we changed pushDir such that the source/local directory is not copied for several cases (notably if adb < 1.0.36, or the remote directory does not exist).

As far as I can tell, 'adb push' will always fail if the source contains symbolic links. The xpcshell test harness (and probably same for mochitest-chrome) push tests from the test staging directory in the objdir, which generally only contains directories and symbolic links back to $topsrcdir.

I think we need to go back to always copying the source directory (or detect links in the source directory and copy if found?).

Comment 5

[Bob Clary [:bc] (inactive)]

I totally missed the symbolic link issue. I think you are right about just doing the copy each time. dir_util.copy_tree will copy the contents of the symbolic links by default as did shutil.copytree. We can get rid of the copyRequired variable and conditional and always remove the temporary source directory.

Do you want to take this?

Comment 6

[Geoff Brown [:gbrown]]

Thanks bc; I think I have this sorted out now.

A promising fix, with extra logging: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b48ab2bac39f1105ec70804e516b9ab2e791217c

Comment 7

[Geoff Brown [:gbrown]]

Attachment: always copy source dir, still allow for adb 1.0.36 and earlier versions

https://treeherder.mozilla.org/#/jobs?repo=try&revision=aaf009e0780e49788b27c12eeae91db7b29bde97

My local testing procedure:

for emulator in (android 4.3, android 6.0)
    for adb in (adb 1.0.32, 1.0.36)
        for test in (xpcshell netwerk/test/unit, mochitest mobile/android/tests/browser/chrome)
            for condition in (clean device without remote directory, unclean device with leftover remote directory)
                run test; while running, check push with 'adb shell ls'

It all works for me!


I wonder if there is still a need to account for the remote directory existing or not. Your comment suggests that there might be, but I can't break this code - which does not check for remote dir existence - with existing tests.

Comment 8

[Bob Clary [:bc] (inactive)]

Comment on attachment 8846789 [details] [diff] [review]
always copy source dir, still allow for adb 1.0.36 and earlier versions

Review of attachment 8846789 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good. I tested adb 1.0.32 and adb 1.0.36 manually and it is fine.

Would you mind doing this to adb.py here and in autophone?

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a9b6d5dfdfa7
Fix devicemanagerADB's pushDir() for adb >= 1.0.36 with links in directory; r=bc

Comment 10

[Geoff Brown [:gbrown]]

(In reply to Bob Clary [:bc:] from comment #8)
> Would you mind doing this to adb.py here and in autophone?

Sure - I'll have a look right away.

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/a9b6d5dfdfa7

Comment 13

[Geoff Brown [:gbrown]]

I left this open for the adb.py change, but now that's happening in bug 1353537.