1486908 - ./mach raptor-test hangs if geckoview_example isn't installed

Status: RESOLVED FIXED

(Testing :: Raptor, defect)

Description

[Bob Clary [:bc] (inactive)]

When I try to run raptor via mach locally without installing geckoview_example first, I get

$ ./mach raptor-test --test raptor-speedometer --app geckoview --binary org.mozilla.geckoview_example 
...
12:37:56     INFO -  adb Ignoring attempt to chmod external storage
12:37:56     INFO -  raptor-main starting geckoview
12:37:56     INFO -  adb launch_application: am start -W -n org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity -a android.intent.action.Main --ez use_multiprocess False --es args '-profile /sdcard/raptor-profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6' -d about:blank
12:37:57     INFO -  adb shell_output: adb -s HT84D1A04087 wait-for-device shell am start -W -n org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity -a android.intent.action.Main --ez use_multiprocess False --es args '-profile /sdcard/raptor-profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6' -d about:blank; echo rc=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: Starting: Intent { act=android.intent.action.Main dat=about:blank cmp=org.mozilla.geckoview_example/.GeckoViewActivity (has extras) }
12:37:57     INFO -  Error type 3
12:37:57     INFO -  Error: Activity class {org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity} does not exist.

which just hangs there. If you wait 1800 seconds, it will time out and everything will be cleaned up.

If I ctrl-c I get

^CProcess Process-1:
12:40:27     INFO - Running post-action listener: _package_coverage_data
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/bclary/mozilla/builds/inbound-taskcluster/mozilla/fennec-opt-arm/testing/raptor-venv/lib/python2.7/site-packages/mozsystemmonitor/resourcemonitor.py", line 90, in _collect
    while not pipe.poll(sleep_interval):
KeyboardInterrupt
12:40:27     INFO - Running post-action listener: _resource_record_post_action
12:40:27     INFO - Running post-action listener: process_java_coverage_data
12:40:27     INFO - [mozharness: 2018-08-28 19:40:27.553235Z] Finished run-tests step (failed)
12:40:27     INFO - Running post-run listener: _resource_record_post_run
12:40:27  WARNING - Exception when reporting resource usage: Traceback (most recent call last):
12:40:27  WARNING -   File "/home/bclary/mozilla/builds/inbound-taskcluster/mozilla/testing/mozharness/mozharness/base/python.py", line 544, in _resource_record_post_run
12:40:27  WARNING -     self._resource_monitor.stop()
12:40:27  WARNING -   File "/home/bclary/mozilla/builds/inbound-taskcluster/mozilla/fennec-opt-arm/testing/raptor-venv/lib/python2.7/site-packages/mozsystemmonitor/resourcemonitor.py", line 285, in stop
12:40:27  WARNING -     self._pipe.send(('terminate',))
12:40:27  WARNING - IOError: [Errno 32] Broken pipe
12:40:27     INFO - Running post-run listener: copy_logs_to_upload_dir
12:40:27     INFO - Copying logs to upload dir...
mach interrupted by signal or user action. Stopping.

which leaves an orphaned process

bclary@ruby ~/mozilla/builds/inbound-taskcluster/mozilla (sisyphus-dev)
$ ps aux | grep raptor.py
bclary   26898  0.1  0.1 474292 25068 pts/2    Sl   12:37   0:00 /home/bclary/mozilla/builds/inbound-taskcluster/mozilla/fennec-opt-arm/testing/raptor-venv/bin/python /home/bclary/mozilla/builds/inbound-taskcluster/mozilla/testing/raptor/raptor/raptor.py --run-local --test raptor-speedometer --app geckoview --binary org.mozilla.geckoview_example --app geckoview --obj-path /home/bclary/mozilla/builds/inbound-taskcluster/mozilla/fennec-opt-arm --log-tbpl-level=debug
bclary   27135  0.0  0.0 213800  1076 pts/2    S+   12:40   0:00 grep --color=auto raptor.py

To get this far you must have first applied the patch in bug 1486892.

I've tried to figure this out but the multiprocess, multithreaded stuff related to resource monitoring and logging has me stumped. The easiest thing I can think of is to quickly fail.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: raptor-fail-if-app-not-installed-or-not-launched.patch

Fail if app isn't installed or the launch fails.

Comment 3

[Geoff Brown [:gbrown]]

In addition to the attached patch (which lgtm), consider adding more user-friendliness at the mach command level. Android mochitest/reftest/xpcshell/junit mach commands use code like:

https://dxr.mozilla.org/mozilla-central/rev/9c13dbdf4cc9baf98881b4e2374363587fb017b7/testing/mozbase/mozrunner/mozrunner/devices/android_device.py#231-236

before starting the test harness.

Comment 4

[Bob Clary [:bc] (inactive)]

gbrown: Thanks. I'll look into that.

Another note about the hang. In bug 1485638 comment 2, I mentioned the problem with the installed version of mozsystemmonitor. I reversed this patch locally, uninstalled geckoview_example and using hacks to force using the in-tree version of mozsystemmonitor I still reproduced the hang but it is the same hang valentin saw in bug 1485638 at https://searchfox.org/mozilla-central/source/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py#101 which sort of confirms the conclusion we also need a fix for that poll().

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: raptor-install-geckoview_example-prompt-locally.patch

Follow up patch to prompt for installation using verify_android_device when running under mach.

Comment 6

[Geoff Brown [:gbrown]]

Comment on attachment 9005021 [details] [diff] [review]
raptor-install-geckoview_example-prompt-locally.patch

Review of attachment 9005021 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome. Glad that worked out.

Comment 7

[Bob Clary [:bc] (inactive)]

Good thing I did a try push first:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=961a82259eedefb0eb5ca88bb0b9c9503acabab2&filter-tier=1&filter-tier=2&filter-tier=3

busted can not import psutil_linux

https://treeherder.mozilla.org/logviewer.html#?job_id=196670238&repo=try&lineNumber=638-667

$ find . -name 'requirements*.txt' | xargs grep psutil
./testing/talos/requirements.txt:psutil>=3.1.1

Including android_device broke mach. Adding psutil to raptor's requirements didn't help.

Comment 8

[Geoff Brown [:gbrown]]

I don't fully understand the problem, but I think you can avoid it by moving the import:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=5da55746dfcdd5443f4ab166ed896e98f28e41bb&filter-tier=1&filter-tier=2&filter-tier=3

https://hg.mozilla.org/try/rev/adf80b64122e5480497e0480f1e0009f6cbf99dd#l1.13

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fb5a461acca4
Raptor should terminate with an exception if the app isn't installed during run_test, r=jmaher.
https://hg.mozilla.org/integration/mozilla-inbound/rev/9a03a046555d
/mach raptor-test should prompt to install geckoview_example if needed, r=gbrown.

Comment 10

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/fb5a461acca4
https://hg.mozilla.org/mozilla-central/rev/9a03a046555d