Closed Bug 1486908 Opened Last year Closed Last year

./mach raptor-test hangs if geckoview_example isn't installed

Categories

(Testing :: Raptor, defect)

defect
Not set

Tracking

(firefox63 fixed)

RESOLVED FIXED
mozilla63
Tracking Status
firefox63 --- fixed

People

(Reporter: bc, Assigned: bc)

References

Details

Attachments

(2 files)

When I try to run raptor via mach locally without installing geckoview_example first, I get

$ ./mach raptor-test --test raptor-speedometer --app geckoview --binary org.mozilla.geckoview_example 
...
12:37:56     INFO -  adb Ignoring attempt to chmod external storage
12:37:56     INFO -  raptor-main starting geckoview
12:37:56     INFO -  adb launch_application: am start -W -n org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity -a android.intent.action.Main --ez use_multiprocess False --es args '-profile /sdcard/raptor-profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6' -d about:blank
12:37:57     INFO -  adb shell_output: adb -s HT84D1A04087 wait-for-device shell am start -W -n org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity -a android.intent.action.Main --ez use_multiprocess False --es args '-profile /sdcard/raptor-profile --es env0 LOG_VERBOSE=1 --es env1 R_LOG_LEVEL=6' -d about:blank; echo rc=$?, timeout: None, root: False, timedout: None, exitcode: 0, output: Starting: Intent { act=android.intent.action.Main dat=about:blank cmp=org.mozilla.geckoview_example/.GeckoViewActivity (has extras) }
12:37:57     INFO -  Error type 3
12:37:57     INFO -  Error: Activity class {org.mozilla.geckoview_example/org.mozilla.geckoview_example.GeckoViewActivity} does not exist.

which just hangs there. If you wait 1800 seconds, it will time out and everything will be cleaned up.

If I ctrl-c I get

^CProcess Process-1:
12:40:27     INFO - Running post-action listener: _package_coverage_data
Traceback (most recent call last):
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 267, in _bootstrap
    self.run()
  File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "/home/bclary/mozilla/builds/inbound-taskcluster/mozilla/fennec-opt-arm/testing/raptor-venv/lib/python2.7/site-packages/mozsystemmonitor/resourcemonitor.py", line 90, in _collect
    while not pipe.poll(sleep_interval):
KeyboardInterrupt
12:40:27     INFO - Running post-action listener: _resource_record_post_action
12:40:27     INFO - Running post-action listener: process_java_coverage_data
12:40:27     INFO - [mozharness: 2018-08-28 19:40:27.553235Z] Finished run-tests step (failed)
12:40:27     INFO - Running post-run listener: _resource_record_post_run
12:40:27  WARNING - Exception when reporting resource usage: Traceback (most recent call last):
12:40:27  WARNING -   File "/home/bclary/mozilla/builds/inbound-taskcluster/mozilla/testing/mozharness/mozharness/base/python.py", line 544, in _resource_record_post_run
12:40:27  WARNING -     self._resource_monitor.stop()
12:40:27  WARNING -   File "/home/bclary/mozilla/builds/inbound-taskcluster/mozilla/fennec-opt-arm/testing/raptor-venv/lib/python2.7/site-packages/mozsystemmonitor/resourcemonitor.py", line 285, in stop
12:40:27  WARNING -     self._pipe.send(('terminate',))
12:40:27  WARNING - IOError: [Errno 32] Broken pipe
12:40:27     INFO - Running post-run listener: copy_logs_to_upload_dir
12:40:27     INFO - Copying logs to upload dir...
mach interrupted by signal or user action. Stopping.

which leaves an orphaned process

bclary@ruby ~/mozilla/builds/inbound-taskcluster/mozilla (sisyphus-dev)
$ ps aux | grep raptor.py
bclary   26898  0.1  0.1 474292 25068 pts/2    Sl   12:37   0:00 /home/bclary/mozilla/builds/inbound-taskcluster/mozilla/fennec-opt-arm/testing/raptor-venv/bin/python /home/bclary/mozilla/builds/inbound-taskcluster/mozilla/testing/raptor/raptor/raptor.py --run-local --test raptor-speedometer --app geckoview --binary org.mozilla.geckoview_example --app geckoview --obj-path /home/bclary/mozilla/builds/inbound-taskcluster/mozilla/fennec-opt-arm --log-tbpl-level=debug
bclary   27135  0.0  0.0 213800  1076 pts/2    S+   12:40   0:00 grep --color=auto raptor.py

To get this far you must have first applied the patch in bug 1486892.

I've tried to figure this out but the multiprocess, multithreaded stuff related to resource monitoring and logging has me stumped. The easiest thing I can think of is to quickly fail.
Fail if app isn't installed or the launch fails.
Attachment #9004661 - Flags: review?(jmaher)
Attachment #9004661 - Flags: review?(jmaher) → review+
Duplicate of this bug: 1484230
Blocks: 1473078
In addition to the attached patch (which lgtm), consider adding more user-friendliness at the mach command level. Android mochitest/reftest/xpcshell/junit mach commands use code like:

https://dxr.mozilla.org/mozilla-central/rev/9c13dbdf4cc9baf98881b4e2374363587fb017b7/testing/mozbase/mozrunner/mozrunner/devices/android_device.py#231-236

before starting the test harness.
gbrown: Thanks. I'll look into that.

Another note about the hang. In bug 1485638 comment 2, I mentioned the problem with the installed version of mozsystemmonitor. I reversed this patch locally, uninstalled geckoview_example and using hacks to force using the in-tree version of mozsystemmonitor I still reproduced the hang but it is the same hang valentin saw in bug 1485638 at https://searchfox.org/mozilla-central/source/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py#101 which sort of confirms the conclusion we also need a fix for that poll().
Depends on: 1485638
Follow up patch to prompt for installation using verify_android_device when running under mach.
Attachment #9005021 - Flags: review?(gbrown)
Comment on attachment 9005021 [details] [diff] [review]
raptor-install-geckoview_example-prompt-locally.patch

Review of attachment 9005021 [details] [diff] [review]:
-----------------------------------------------------------------

Awesome. Glad that worked out.
Attachment #9005021 - Flags: review?(gbrown) → review+
Good thing I did a try push first:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=961a82259eedefb0eb5ca88bb0b9c9503acabab2&filter-tier=1&filter-tier=2&filter-tier=3

busted can not import psutil_linux

https://treeherder.mozilla.org/logviewer.html#?job_id=196670238&repo=try&lineNumber=638-667

$ find . -name 'requirements*.txt' | xargs grep psutil
./testing/talos/requirements.txt:psutil>=3.1.1

Including android_device broke mach. Adding psutil to raptor's requirements didn't help.
Flags: needinfo?(gbrown)
Pushed by bclary@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fb5a461acca4
Raptor should terminate with an exception if the app isn't installed during run_test, r=jmaher.
https://hg.mozilla.org/integration/mozilla-inbound/rev/9a03a046555d
/mach raptor-test should prompt to install geckoview_example if needed, r=gbrown.
https://hg.mozilla.org/mozilla-central/rev/fb5a461acca4
https://hg.mozilla.org/mozilla-central/rev/9a03a046555d
Status: ASSIGNED → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Depends on: 1488544
Blocks: 1483695
You need to log in before you can comment on or make changes to this bug.