1466657 - [mozdevice] get_process_list sometimes fails (Android x86 7.0 emulator)

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Geoff Brown [:gbrown]]

In bug 1460411, when I run mochitests on the Android x86 7.0 emulator, get_process_list intermittently fails. It looks like 'ps' returns no output. If I repeat the request, it usually succeeds.

Comment 1

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/logviewer.html#?job_id=180832250&repo=try&lineNumber=1101-1114

[task 2018-05-30T03:57:50.045Z] 03:57:50     INFO -  0 ERROR adb get_process_list:
[task 2018-05-30T03:57:50.045Z] 03:57:50     INFO -  Traceback (most recent call last):
[task 2018-05-30T03:57:50.045Z] 03:57:50     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtests.py", line 2802, in doTests
[task 2018-05-30T03:57:50.045Z] 03:57:50     INFO -      marionette_args=marionette_args,
[task 2018-05-30T03:57:50.046Z] 03:57:50     INFO -    File "/builds/worker/workspace/build/tests/mochitest/runtestsremote.py", line 321, in runApp
[task 2018-05-30T03:57:50.046Z] 03:57:50     INFO -      ret, _ = self.automation.runApp(*args, **kwargs)
[task 2018-05-30T03:57:50.046Z] 03:57:50     INFO -    File "/builds/worker/workspace/build/tests/mochitest/automation.py", line 531, in runApp
[task 2018-05-30T03:57:50.046Z] 03:57:50     INFO -      self.log.info("INFO | automation.py | Application pid: %d", proc.pid)
[task 2018-05-30T03:57:50.046Z] 03:57:50     INFO -    File "/builds/worker/workspace/build/tests/mochitest/remoteautomation.py", line 278, in pid
[task 2018-05-30T03:57:50.046Z] 03:57:50     INFO -      procs = self.device.get_process_list()
[task 2018-05-30T03:57:50.047Z] 03:57:50     INFO -    File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 1994, in get_process_list
[task 2018-05-30T03:57:50.047Z] 03:57:50     INFO -      header, adb_process))
[task 2018-05-30T03:57:50.047Z] 03:57:50 CRITICAL -  ADBError: get_process_list: Unknown format: : args: adb -s emulator-5554 wait-for-device shell ps; echo rc=$?, exitcode: None, stdout:
[task 2018-05-30T03:57:50.047Z] 03:57:50     INFO -  1 ERROR Automation Error: Received unexpected exception while running application

Comment 2

[Geoff Brown [:gbrown]]

More examples:

https://treeherder.mozilla.org/logviewer.html#?job_id=180832282&repo=try&lineNumber=3011-3023
https://treeherder.mozilla.org/logviewer.html#?job_id=180832286&repo=try&lineNumber=2950-2962
https://treeherder.mozilla.org/logviewer.html#?job_id=180832297&repo=try&lineNumber=3229-3242

Comment 3

[Geoff Brown [:gbrown]]

I added some debugging and retry logic:

https://hg.mozilla.org/try/rev/4e46e9219604c710f7ece9bd2f9a69281d40b013

and the retry seems to work:

https://treeherder.mozilla.org/logviewer.html#?job_id=181769739&repo=try&lineNumber=2191-2200

Comment 4

[Geoff Brown [:gbrown]]

:bc - Any thoughts? Retry when an empty/invalid header is returned from ps? Retry in get_process_list, or in the caller?

Comment 5

[Geoff Brown [:gbrown]]

(Wish you were here!)

Comment 6

[Bob Clary [:bc] (inactive)]

I would retry once in get_process_list. I wonder what is wrong with the emulator. I noticed a lot of messages about chatty dropping lines. We handled this in Autophone in bug 1360920. I wonder if that is related.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: retry 'ps' in get_process_list

Seems harmless:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e30d7c4f2b8ab1c6c877099dd9b5c38a67a7c77

Comment 8

[Bob Clary [:bc] (inactive)]

Comment on attachment 8985563 [details] [diff] [review]
retry 'ps' in get_process_list

Review of attachment 8985563 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with style nit.

::: testing/mozbase/mozdevice/mozdevice/adb.py
@@ +2003,5 @@
> +                        continue
> +                    raise ADBError('get_process_list: Unknown format: %s: %s' % (
> +                        header, adb_process))
> +                else:
> +                    break

This is a bit more readable for me:

                if user_i != -1 and pid_i != -1:
                    break
                self._logger.error('get_process_list: %s' % header)
                if attempt >= max_attempts:
                    raise ADBError('get_process_list: Unknown format: %s: %s' %
                                   (header, adb_process))

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8bd89325bde7
Retry Android 'ps' command on unexpected output; r=bc

Comment 10

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/8bd89325bde7