1199377 - MOZ_HOST_BIN setup should be automated

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

Obtaining host binaries and setting up MOZ_HOST_BIN for Android testing is documented at https://wiki.mozilla.org/Mobile/Fennec/Android/Testing#Host_Builds_.28MOZ_HOST_BIN.29. If not set correctly, mach commands fail with helpful error messages. Still, this is an additional barrier and (one time) bother to anyone running tests on Android. In the make-it-dead-simple spirit of bug 1164596, let's try to simplify or eliminate this requirement.

Comment 1

[Geoff Brown [:gbrown]]

Complication: mozharness jobs pull host_utils from http://talos-remote.pvt.build.mozilla.org -- not visible to many developers; but a copy is available on tooltool now. Why doesn't mozharness pull from tooltool?

Complication: host_utils binaries must match the desktop architecture -- we need multiple tooltool entries.

Comment 2

[(not currently active) Ted Mielczarek]

This feels very related to the work nalexander was doing for pulling down pre-built Gecko bits.

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #2)
> This feels very related to the work nalexander was doing for pulling down
> pre-built Gecko bits.

Mmm, I agree, although the actual technical implementation probably has little bearing.  See also Bug 1126424, which tracks making mach do this (in some form).

I'd be happy to make this part of |mach bootstrap|, except it has been historically hard to ensure |mach bootstrap| keeps working (since it has no automatic tests) and the host binaries are bloat that many folks will not need.

Comment 4

[Geoff Brown [:gbrown]]

I don't want to slow down bootstrap for those who don't need host-utils.

I think a dedicated mach command is overkill. There are so many mach commands now...I don't want to add another command that will be forgotten and rarely used.

Instead, I am thinking we can download hostutils as needed at . For example:

  $ mach robocop
  MOZ_HOST_BIN is not set in your environment. Download and setup your host-utils? (Y/n)
  Downloading host-utils. This may take a while....

I am trying to do something similar for the emulator images and apk in bug 1164596. So a first-time test run might look like:

  $ mach robocop
  No Android devices connected. Start an emulator? (Y/n) 
  Fetching AVD. This may take a while...
  Starting Android emulator running Android 4.3...
  It looks like Firefox is not installed on this device.
  Install Firefox? (Y/n) 
  Installing Firefox. This may take a while...
  MOZ_HOST_BIN is not set in your environment. Download and setup your host-utils? (Y/n)
  Downloading host-utils -- this may take a while...
From _tests: Kept 36262 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.00 LOG: MainThread INFO Checking for orphan ssltunnel processes...
 0:00.07 LOG: MainThread INFO Checking for orphan xpcshell processes...
 0:01.46 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests
pk12util: PKCS12 IMPORT SUCCESSFUL

Comment 5

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #4)
> Instead, I am thinking we can download hostutils as needed at . For example:

...as needed at test time (when a test is run but host-utils cannot be found)...

Comment 6

[Geoff Brown [:gbrown]]

Attachment: add tooltool manifests for host-utils

I have uploaded the host-utils from https://wiki.mozilla.org/Mobile/Fennec/Android/Testing#Quick_setup to tooltool. These were validated in bug 1126424 (plus I verified the linux64 version myself just today).

One day I'll update the packages with a recent build and update mozharness to pull from tooltool, but there's no pressing need for that.

This patch adds the associated tooltool manifests. I'll attach my wip patch showing how I intend to use these from mach.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: wip

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8660969 [details] [diff] [review]
add tooltool manifests for host-utils

Review of attachment 8660969 [details] [diff] [review]:
-----------------------------------------------------------------

the only question here is when we update hostutils, what do we do?  Do we have to update all 3 of these files?  Also is .xz the proper extension?

Comment 9

[Geoff Brown [:gbrown]]

Yes, when we update hostutils, in most cases we will want to update all supported platforms, so all 3 files will need to be updated. 

The archives are xz compressed -- https://en.wikipedia.org/wiki/XZ_Utils -- which is well supported generally and explicitly by tar, so tooltool should be happy with it.

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b33de7c40755

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/b33de7c40755

Comment 12

[Geoff Brown [:gbrown]]

Attachment: download host-utils from tooltool when necessary

This treats host-utils very much like an avd or Firefox installation. If host-utils are not found via MOZ_HOST_BIN and not installed in .mozbuild, download from tooltool, and install in .mozbuild.

The host-utils check is required for mochitests and robocop but not for xpcshell or cppunittests; once mach supports reftests on Android, those will also need to check for host-utils.

Since verify_android_device() needed some "private" functions from AndroidEmulator, I moved some functions out of AndroidEmulator. I could have kept them in AndroidEmulator but on reflection decided these were not closely related to the emulator and belonged elsewhere.

gbrown@mozpad:~/src$ rm -r ~/.mozbuild/android-device
gbrown@mozpad:~/src$ export MOZ_HOST_BIN=
gbrown@mozpad:~/src$ ./mach robocop testLoad
No Android devices connected. Start an emulator? (Y/n) y
Fetching AVD. This may take a while...
Starting emulator running Android 4.3...
It looks like Firefox is not installed on this device.
Install Firefox? (Y/n) y
Installing Firefox. This may take a while...
Host utilities not found: environment variable MOZ_HOST_BIN is not set to a directory containing host xpcshell
Download and setup your host utilities? (Y/n) y
Downloading host utilities -- this may take a while...
From _tests: Kept 36374 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.98 LOG: MainThread INFO Checking for orphan ssltunnel processes...
 0:01.06 LOG: MainThread INFO Checking for orphan xpcshell processes...
 0:02.45 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests
pk12util: PKCS12 IMPORT SUCCESSFUL
 0:09.98 LOG: MainThread INFO MochitestServer : launching [u'/home/gbrown/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64/xpcshell', '-g', '/home/gbrown/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64', '-v', '170', '-f', '/home/gbrown/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpMJ5bwC.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '192.168.0.82'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/home/gbrown/objdirs/droid/_tests/testing/mochitest/server.js']
 0:09.98 LOG: MainThread INFO runtests.py | Server pid: 23773
 0:09.99 LOG: MainThread INFO runtests.py | Websocket server pid: 23776
 0:10.01 LOG: MainThread INFO runtests.py | SSL tunnel pid: 23781
 0:11.01 SUITE_START: MainThread 1
INFO | automation.py | Application pid: 0
 0:12.89 LOG: None INFO SimpleTest START
 0:12.89 TEST_START: None testLoad
 0:51.45 LOG: MainThread INFO EventExpecter: no longer listening for Gecko:Ready
 0:51.45 LOG: MainThread INFO EventExpecter: no longer listening for DOMContentLoaded
 0:51.45 LOG: MainThread INFO PaintExpecter: no longer listening for events
 0:47.19 TEST_END: None Harness OK. Subtests passed 8/8. Unexpected 0
 0:47.20 LOG: None INFO TEST-START | Shutdown
 0:47.20 LOG: None INFO Passed: 8
 0:47.20 LOG: None INFO Failed: 0
 0:47.21 LOG: None INFO Todo: 0
 0:47.21 LOG: None INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:00:50.161554
INFO | zombiecheck | Reading PID log: /tmp/tmpUhmZcgpidlog
MOZ_UPLOAD_DIR not defined; tombstone check skipped
INFO | runtests.py | Test summary: start.
0 INFO TEST-START | Shutdown
1 INFO Passed: 8
2 INFO Failed: 0
3 INFO Todo: 0
4 INFO SimpleTest FINISHED
INFO | runtests.py | Test summary: end.
 1:04.00 LOG: MainThread INFO Stopping web server
 1:04.03 LOG: MainThread INFO Stopping web socket server
 1:04.05 LOG: MainThread INFO Stopping ssltunnel
 1:05.71 SUITE_END: MainThread 
Summary
=======

Ran 9 tests (1 parents, 8 subtests)
Expected results: 9
Unexpected results: 0

OK
gbrown@mozpad:~/src$ ./mach robocop testLoad
No Android devices connected. Start an emulator? (Y/n) y
Starting emulator running Android 4.3...
From _tests: Kept 36374 existing; Added/updated 0; Removed 0 files and 0 directories.
 0:00.97 LOG: MainThread INFO Checking for orphan ssltunnel processes...
 0:01.02 LOG: MainThread INFO Checking for orphan xpcshell processes...
 0:02.21 LOG: MainThread INFO Android sdk version '18'; will use this to filter manifests
pk12util: PKCS12 IMPORT SUCCESSFUL
 0:10.85 LOG: MainThread INFO MochitestServer : launching [u'/home/gbrown/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64/xpcshell', '-g', '/home/gbrown/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64', '-v', '170', '-f', '/home/gbrown/.mozbuild/android-device/host-utils-37.0a2.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpIyE8hC.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '192.168.0.82'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/home/gbrown/objdirs/droid/_tests/testing/mochitest/server.js']
 0:10.85 LOG: MainThread INFO runtests.py | Server pid: 24303
 0:10.86 LOG: MainThread INFO runtests.py | Websocket server pid: 24306
 0:10.87 LOG: MainThread INFO runtests.py | SSL tunnel pid: 24311
 0:11.87 SUITE_START: MainThread 1
INFO | automation.py | Application pid: 0
 0:13.90 LOG: None INFO SimpleTest START
 0:13.91 TEST_START: None testLoad
 0:51.67 LOG: MainThread INFO EventExpecter: no longer listening for Gecko:Ready
 0:51.67 LOG: MainThread INFO EventExpecter: no longer listening for DOMContentLoaded
 0:51.68 LOG: MainThread INFO PaintExpecter: no longer listening for events
 0:47.53 TEST_END: None Harness OK. Subtests passed 8/8. Unexpected 0
 0:47.54 LOG: None INFO TEST-START | Shutdown
 0:47.54 LOG: None INFO Passed: 8
 0:47.55 LOG: None INFO Failed: 0
 0:47.55 LOG: None INFO Todo: 0
 0:47.55 LOG: None INFO SimpleTest FINISHED
INFO | automation.py | Application ran for: 0:00:48.665458
INFO | zombiecheck | Reading PID log: /tmp/tmpqHrVw_pidlog
/data/anr/traces.txt not found
MOZ_UPLOAD_DIR not defined; tombstone check skipped
INFO | runtests.py | Test summary: start.
0 INFO TEST-START | Shutdown
1 INFO Passed: 8
2 INFO Failed: 0
3 INFO Todo: 0
4 INFO SimpleTest FINISHED
INFO | runtests.py | Test summary: end.
 1:04.00 LOG: MainThread INFO Stopping web server
 1:04.03 LOG: MainThread INFO Stopping web socket server
 1:04.05 LOG: MainThread INFO Stopping ssltunnel
 1:05.71 SUITE_END: MainThread 
Summary
=======

Ran 9 tests (1 parents, 8 subtests)
Expected results: 9
Unexpected results: 0

OK

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8661990 [details] [diff] [review]
download host-utils from tooltool when necessary

Review of attachment 8661990 [details] [diff] [review]:
-----------------------------------------------------------------

just one small nit!

::: testing/mozbase/mozrunner/mozrunner/devices/android_device.py
@@ +144,5 @@
> +        elif not os.path.isdir(xre_path):
> +            err = '$MOZ_HOST_BIN does not specify a directory'
> +        elif not os.path.isfile(os.path.join(xre_path, 'xpcshell')):
> +            err = '$MOZ_HOST_BIN/xpcshell does not exist'
> +        if err is not None:

this is confusing, I would rather see:
if err:

Comment 14

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/1dbf28047168

Comment 15

[Geoff Brown [:gbrown]]

Thanks. Simplified x3 on landing.

Comment 16

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/1dbf28047168