1143834 - [mozrunner] Support running B2G on a remote device

Status: RESOLVED WONTFIX

(Testing :: Mozbase, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Currently mozrunner assumes that any devices are attached to your local machine. We now have a remote device lab, so we need to update mozrunner to allow us to specify the remote ADB and Marionette servers.

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Support connecting to a remote ADB server. v1.0

Here's my first stab at this. It's working for me locally using a modified b2gmonkey. Note that this depends on bug 1140406, which also has a very early version of a patch.

Here's a try run with both patches. If nothing breaks I'll be amazed: https://treeherder.mozilla.org/#/jobs?repo=try&revision=70a38773bfb2

Comment 2

[Andrew Halberstadt [:ahal]]

Comment on attachment 8578224 [details] [diff] [review]
Support connecting to a remote ADB server. v1.0

Review of attachment 8578224 [details] [diff] [review]:
-----------------------------------------------------------------

This approach looks good to me!

::: testing/mozbase/mozrunner/mozrunner/application.py
@@ +50,5 @@
>          self._adb = adb_path
>          self._adb_host = adb_host
>          self._adb_port = adb_port
> +        self._marionette_host = marionette_host
> +        self._marionette_port = marionette_port

Currently there's no assumption that mozrunner needs marionette (except a couple methods that get a marionette object passed in), which I think would be a nice property to preserve. It looks like `marionette_host` is only used in wait_for_port().. could we maybe add a 'host' parameter to that method and call it with the proper host from whatever automation is setting this up instead?

::: testing/mozbase/mozrunner/mozrunner/devices/base.py
@@ +196,5 @@
>          starttime = datetime.datetime.now()
>          while datetime.datetime.now() - starttime < datetime.timedelta(seconds=timeout):
>              try:
>                  sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
> +                sock.connect((self.app_ctx.marionette_host, port))

This should be _marionette_host, but I'd rather this function not be marionette specific.

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Attachment: Support connecting to a remote ADB server. v2.0

Latest version of the patch, still a work in progress.

Comment 4

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've managed to address a lot of the try failures, but there are a few that still elude me. Here's the latest try run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4421b6d17e06&exclusion_profile=false

Jonathan: Any guidance you can provide would be very helpful. This patch appears to have caused the following issues.

Mn: test_elementsize.py consistently fails while appearing to try to switch to a newly opened window. I can't see how my patch could cause this...

Mnw: Three tests failing due to "Unexpected number of running processes:hostapd, expected: 3, actual: 4" where expected always appears to be actual-1. This is followed by a timeout trying to remount during cleanup, which certainly is caused by my patch - adb.py differs to devicemanagerADB.py in that commands first wait for the device to be ready. I suspect in this case the emulator has already been killed. I could guard this cleanup by first checking if the device is ready (get_state == 'device') with a low timeout, or I could catch the ADBTimeoutError on the remount call. Do you have any suggestions?

X & X2: Looks like these time out and don't actually run any tests. :(

M(9&20): This appears to be similar to the above mentioned ADBTimeoutError when running cleanup. It's trying to pull a log file from the emulator, but I suspect the emulator is already terminated. I'm a little unsure why we're running cleanup if the emulator has stopped, and wonder if that's indicating another issue, especially as the other mochitests do not suffer from this.

M(c): This looks like a crash that's not handled well (possibly a separate bug). I could do with some pointers here.

R(C1): Looks like more crashes. Perhaps my patch breaks the detection of crashes?

M(12): This looks like bug 1099195.

I can keep digging, but I thought it might be worth asking in case anything here looks familiar or if there's anywhere in particular I should focus. Thanks!

Comment 5

[Jonathan Griffin (:jgriffin)]

The Mn failures aren't yours, they exist in m-c too.  But, the difference is that with your patch, it seems that the emulator is being restarted after the timeout, which isn't desirable.

This same factor seems to be at issue with the mochitest failures...on failure we can't pull the log, and I agree it seems to be that the emulator has already been terminated, and subsequently restarted, since the emulator log is very brief:  http://mozilla-releng-blobs.s3.amazonaws.com/blobs/try/sha512/6b01337772674e814c5b72ab583a0503becf418cf916814589f6871eff555b5c74b9f6c1948b70db113844e5f6619e44e94125cc03a79e5e5dc8cb46056c0eef

As you suggest, I suspect this is breaking crash detection.

I'd try to track down why we restart the emulator on failure, and fix that...a lot of this might go away then.

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I'm no longer working on this.