810546 - dmADB's pushDir could be faster if zip not installed

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[William Lachance (:wlach)]

In a conversation I had with :dmose and :wesj, it came up the dmADB's pushDir method (used by pretty much all the tests to copy profile/test information) is pretty unbearably slow if zip is not installed on the device. This is because we currently copy things file-by-file to get around the fact that pushing an entire directory copies symbolic links to the device literally, which is not what we want. But we can get around that by copying the directory structure to a tempdir (dereferencing things as required) and then pushing *that* to the device.

Comment 1

[William Lachance (:wlach)]

Attachment: Speed up pushdir when zip not installed

In my quick testing of pushing the mozdevice source dir to a device without zip installed, I see that the time needed went from 13 seconds to 3 seconds. Not bad.

I also verified that symbolic links are still copied over correctly:
wlach@popsicle:~/src/mozilla-central$ mkdir /tmp/testdir
wlach@popsicle:~/src/mozilla-central$ echo "cheezburgers are fun" > /tmp/testdir/file1
wlach@popsicle:~/src/mozilla-central$ ln -s /tmp/testdir/file1 /tmp/testdir/file2
wlach@popsicle:~/src/mozilla-central$ ls -al /tmp/testdir/
total 68
drwxrwxr-x  2 wlach wlach  4096 Nov  9 18:30 .
drwxrwxrwt 53 root  root  61440 Nov  9 18:30 ..
-rw-rw-r--  1 wlach wlach    21 Nov  9 18:30 file1
lrwxrwxrwx  1 wlach wlach    18 Nov  9 18:30 file2 -> /tmp/testdir/file1

(mozbase)wlach@popsicle:~/src/mozbase$ dm push /tmp/testdir /mnt/sdcard/testdirAndroid Debug Bridge version 1.0.29

push: /tmp/tmpbP6UF7/tmp/file1 -> /mnt/sdcard/testdir/file1
push: /tmp/tmpbP6UF7/tmp/file2 -> /mnt/sdcard/testdir/file2
2 files pushed. 0 files skipped.
0 KB/s (42 bytes in 0.087s)
(mozbase)wlach@popsicle:~/src/mozbase$ adb shell
shell@android:/ $ ls -al /mnt/sdcard/testdir/file1
-rw-rw-r-- root     sdcard_rw       21 2012-11-09 18:42 file1
shell@android:/ $ ls -al /mnt/sdcard/testdir/file2                             
-rw-rw-r-- root     sdcard_rw       21 2012-11-09 18:42 file2

Comment 2

[Geoff Brown [:gbrown]]

Comment on attachment 680305 [details] [diff] [review]
Speed up pushdir when zip not installed

Wicked! I love it.

shutil.copytree is new in python 2.6 -- is that okay?

Comment 3

[Dan Mosedale (:dmosedale, :dmose)]

Wonderful; thanks for pushing this forward, William!  Everything that makes it cheaper to run tests will make it easier for people to embrace development workflows that are focused on testing (like TDD).

Comment 4

[William Lachance (:wlach)]

(In reply to Geoff Brown [:gbrown] from comment #2)
> Comment on attachment 680305 [details] [diff] [review]
> Speed up pushdir when zip not installed
> 
> Wicked! I love it.
> 
> shutil.copytree is new in python 2.6 -- is that okay?

It looks like copytree dates back to before 2.3, though its behaviour has changed slightly over the versions (fortunately in ways that shouldn't affect us):

http://docs.python.org/2/library/shutil.html#shutil.copytree

Comment 5

[Geoff Brown [:gbrown]]

(In reply to William Lachance (:wlach) from comment #4)
> It looks like copytree dates back to before 2.3, though its behaviour has
> changed slightly over the versions (fortunately in ways that shouldn't
> affect us):

Sorry, I confused the copytree description with the one above it -- you are right / no issue here.

Comment 6

[William Lachance (:wlach)]

Pushed: https://github.com/mozilla/mozbase/commit/550f453009f5cfa686ff7800c92cc9f37203d02a

(note that we will need to merge mozbase into m-c to see these changes... planning on doing that this week though)

Comment 7

[Dan Mosedale (:dmosedale, :dmose)]

Very cool.  Is that merge tracked by a bug that I can follow?

Comment 8

[William Lachance (:wlach)]

(In reply to Dan Mosedale (:dmose) from comment #7)
> Very cool.  Is that merge tracked by a bug that I can follow?

Not yet. There's a few other patches I'd like to land before doing a release hopefully later today and doing the merge. I'll be sure to CC you when I file a bug for that.