1123443 - Allow android emulator tests to use adb devicemanager

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

We currently use sut devicemanager for all android emulator unit tests (Android 2.3 and Android x86), and android_emulator_unittest.py assumes that sutagent is installed and running on the emulator.

In bug 1062365, we might want to use adb instead of sut for emulators running Android 4.4 and higher.

I think we should at least have the option: use either sut or adb for tests started by android_emulator_unittest.py.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: wip - mozharness changes

Comment 2

[Geoff Brown [:gbrown]]

Attachment: wip - in-tree config update

Comment 3

[Geoff Brown [:gbrown]]

Attachment: update to in-tree config -- make --dm-trans more flexible

Comment 4

[Geoff Brown [:gbrown]]

Attachment: mozharness changes -- add "device_manager" to config and support in android_emulator_unittest

This adds a "device_manager" parameter to the android emulator configuration. android_emulator_unittest.py is updated to:
 - not redirect SUT when adb is configured
 - not verify SUT connection when adb is configured
 - not pass --deviceIP and --devicePort in-tree options when adb is configured
Existing configuration files for Android 2.3 and Android x86 are configured for sut, so no change is expected to existing tests.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9bf7b007068b verifies this does no harm to existing tests. (The gl1 retry is an existing condition currently seen on mozilla-central.)

Comment 5

[Geoff Brown [:gbrown]]

Comment on attachment 8551457 [details] [diff] [review]
update to in-tree config -- make --dm-trans more flexible

This updates the in-tree configuration to allow the --dm_trans option to pick up the devicemanager configured in mozharness.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=6dd050cbc39d confirms that existing tests keep running fine and using sut. 

The 2nd mochitest-1 run, http://ftp.mozilla.org/pub/mozilla.org/mobile/try-builds/[e-mail]-6dd050cbc39d/try-android-api-9/try_ubuntu64_vm_mobile_test-mochitest-1-bm115-tests1-linux64-build165.txt.gz, uses adb -- and still works!

Comment 6

[Armen [:armenzg]]

Comment on attachment 8551876 [details] [diff] [review]
mozharness changes -- add "device_manager" to config and support in android_emulator_unittest

Review of attachment 8551876 [details] [diff] [review]:
-----------------------------------------------------------------

r- for the modification of options on the fly.
I believe we can fix this by specifying an in-tree config for Android 4.4 jobs where we specify adb and we don't specify --deviceIP and --devicePort.

::: scripts/android_emulator_unittest.py
@@ +430,5 @@
>          for option in self.tree_config["suite_definitions"][suite_category]["options"]:
> +            # remove --deviceIP and --devicePort if using adb devicemanager
> +            if not (c['device_manager'] == 'adb' and
> +                    (option.startswith('--deviceIP') or option.startswith('--devicePort'))):
> +                cmd.extend([option % str_format_values])

I'm not fond of modifying options on the fly.

I would like us to isolate Android 2.3 jobs from Android 4.4 jobs definitions.
I think we should add an in-tree config for Android 4.4 jobs where we specify to use adb.

Would this work for you?

Comment 7

[Geoff Brown [:gbrown]]

I agree that modifying options on the fly is bad.

I agree that we need to add an in-tree config for Android 4.4.

I would like to be able to change a job from sut to adb or vice versa -- and that is complicated because --deviceIP and --devicePort are tied to --dm_trans, which is tied to the sut validation behavior of android_emulator_unittest.py. We could support "--deviceIP=%(device_ip)s" but I think we would also need to come up with a way to specify, "if %(device_ip)s is null, do not add --deviceIP", which is kind-of an advanced form of modifying options on the fly!

I'll try adding in-tree config for Android 4.4 for now and see how that goes.

Comment 8

[Geoff Brown [:gbrown]]

Attachment: in-tree config for android 4.4

Comment 9

[Geoff Brown [:gbrown]]

Attachment: wip - mozharness changes: add "device_manager" to config, add android 4.4 config

Comment 10

[Armen [:armenzg]]

(In reply to Geoff Brown [:gbrown] from comment #7)
> I would like to be able to change a job from sut to adb or vice versa -- and
> that is complicated because --deviceIP and --devicePort are tied to
> --dm_trans, which is tied to the sut validation behavior of
> android_emulator_unittest.py. We could support "--deviceIP=%(device_ip)s"
> but I think we would also need to come up with a way to specify, "if
> %(device_ip)s is null, do not add --deviceIP", which is kind-of an advanced
> form of modifying options on the fly!
> 
The joy of programming :)

Comment 11

[Geoff Brown [:gbrown]]

Comment on attachment 8551987 [details] [diff] [review]
in-tree config for android 4.4

Review of attachment 8551987 [details] [diff] [review]:
-----------------------------------------------------------------

Here's an in-tree config for 4.4. This is a copy of android_arm_config.py (the Android 2.3 config) with --dm_trans="adb" and --deviceIP and --devicePort arguments removed.

Comment 12

[Geoff Brown [:gbrown]]

Attachment: mozharness changes: add "device_manager" to config, add android 4.4 config

Add a 4.4 mozharness config to reference the 4.4 in-tree config.
Add "device_manager" to android configs.
Avoid sut redirection and other sut references when adb configured.

No harm done: https://treeherder.mozilla.org/#/jobs?repo=try&revision=e9e08efa8157

Comment 13

[Armen [:armenzg]]

Comment on attachment 8552158 [details] [diff] [review]
mozharness changes: add "device_manager" to config, add android 4.4 config

Review of attachment 8552158 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/android/androidarm_4_4.py
@@ +11,5 @@
> +    "emulator_process_name": "emulator64-arm",
> +    "emulator_cpu": "cortex-a9",
> +    "device_manager": "adb",
> +    "exes": {
> +        'adb': '/tools/android-sdk18/platform-tools/adb',

I assume this (and similar below) will eventually change.

::: scripts/android_emulator_unittest.py
@@ +428,5 @@
> +        if self.config["device_manager"] == "sut":
> +            str_format_values.extend({
> +                'device_ip': c['device_ip'],
> +                'device_port': str(emulator['sut_port1']),
> +            })

It works for me :)

Comment 14

[Geoff Brown [:gbrown]]

(In reply to Armen Zambrano - Automation & Tools Engineer (:armenzg) from comment #13)
> I assume this (and similar below) will eventually change.

Yes - I'll modify the 4.4 config incrementally in other bugs.

Comment 15

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/447467c0b2c4

Comment 16

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/build/mozharness/rev/313d91e863f5

Comment 17

[Geoff Brown [:gbrown]]

Here is one more incremental test showing Android 2.3 running with adb instead of sut: https://treeherder.mozilla.org/#/jobs?repo=try&revision=470476640967

Most tests run fine, but there are a few failures that look new and may well be related to the devicemanager change. I won't investigate these failures since we have no plans to run this configuration, but these results might be interesting once 4.4 is stood up.

Comment 18

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/447467c0b2c4

Comment 19

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/313d91e863f5

Comment 20

[Chris Cooper [:coop] (he/him)]

(In reply to Chris Cooper [:coop] from comment #19)
> In production: https://hg.mozilla.org/build/mozharness/rev/313d91e863f5

Backed out for failures like this:

https://treeherder.mozilla.org/logviewer.html#?job_id=923324&repo=mozilla-central

Comment 21

[Geoff Brown [:gbrown]]

s/extend/update/ seems to do the trick, but I will verify (again, more carefully) on try before pushing again.

Comment 22

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=262e7fcf08f3 looks solid.

https://hg.mozilla.org/build/mozharness/rev/cd253454cf3c

Comment 23

[Armen [:armenzg]]

https://hg.mozilla.org/build/ash-mozharness/rev/6bd3653426f8

Comment 24

[Jordan Lund (:jlund)]

a mozharness patch has from this bug is in production