701038 - use zip/unzip to optimize devicemanagerADB's pushDir

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

When running xpcshell-tests-remote, devicemanagerADB must push the entire xpcshell tests directory (many hundreds of small files) to device. It does this one file at a time, and this typically takes over 30 minutes.

If zip is available on the local host and unzip is available on the remote host (unzip seems to be included in busybox), then pushDir can zip the directory, push just one zip file, and unzip on the device. Preliminary results suggest this can be done for xpcshell-tests-remote in under 4 minutes (unzipping takes most of the time).

Comment 1

[Geoff Brown [:gbrown]]

Attachment: patch to zip/unzip in pushDir

A preliminary patch -- needs more testing.

Comment 2

[cmtalbert]

(In reply to Geoff Brown [:gbrown] from comment #0)
> When running xpcshell-tests-remote, devicemanagerADB must push the entire
> xpcshell tests directory (many hundreds of small files) to device. It does
> this one file at a time, and this typically takes over 30 minutes.
> 
> If zip is available on the local host and unzip is available on the remote
> host (unzip seems to be included in busybox), then pushDir can zip the
> directory, push just one zip file, and unzip on the device. Preliminary
> results suggest this can be done for xpcshell-tests-remote in under 4
> minutes (unzipping takes most of the time).

One drive-by comment:
I know that we have unzip implemented in the SUTagent, not sure if that helps you at all though if your target is to use the ADB agent.  But it might be worth looking at how that is implemented.

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Clint Talbert ( :ctalbert ) from comment #2)
> One drive-by comment:
> I know that we have unzip implemented in the SUTagent, not sure if that
> helps you at all though if your target is to use the ADB agent.  But it
> might be worth looking at how that is implemented.

Thanks for the tip! I found that devicemanagerSUT.unpackFile uses the "unzp" command. On the agent, Java classes from java.util.zip are used to unzip the stream. I won't use this for adb, but have spun off bug 705162 for SUT.

Comment 4

[Geoff Brown [:gbrown]]

Attachment: patch to zip/unzip in ADB pushDir

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 578972 [details] [diff] [review]
patch to zip/unzip in ADB pushDir

Review of attachment 578972 [details] [diff] [review]:
-----------------------------------------------------------------

this looks great!

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/232990286849

Comment 7

[Matt Brubeck (:mbrubeck)]

https://hg.mozilla.org/mozilla-central/rev/232990286849