1655164 - Perma Android Btime-live-ChR failures

Status: RESOLVED FIXED

(Testing :: Raptor, defect, P2)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

First failure on Monday: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedTaskRun=cDCDj53GQIWnfvRMbzFB_A.0&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel%2Crunning%2Cpending%2Crunnable&tier=1%2C2%2C3&revision=2c0c87130a4975ebd069846e1dc9b9b02c3b4aaa

Log: https://treeherder.mozilla.org/logviewer.html#?job_id=310370430&repo=mozilla-central

[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - [mozharness: 2020-07-20 12:12:28.379938Z] Running install-chrome-android step.
[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - Running pre-action listener: _resource_record_pre_action
[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - Running main action method: install_chrome_android
[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - Fetching and installing Google Chrome for Android
[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - retry: Calling run_command with args: (['/usr/bin/python2.7', '-u', '/builds/task_1595246656/workspace/mozharness/external_tools/tooltool.py', 'fetch', '-m', u'/builds/task_1595246656/workspace/build/tests/raptor/raptor/tooltool-manifests/chrome-android/chrome80.manifest', '-o'],), kwargs: {'output_timeout': 600, 'error_list': [{'substr': 'command not found', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f0cb2a23180>, 'level': 'warning'}, {'substr': 'Traceback (most recent call last)', 'level': 'error'}, {'substr': 'SyntaxError: ', 'level': 'error'}, {'substr': 'TypeError: ', 'level': 'error'}, {'substr': 'NameError: ', 'level': 'error'}, {'substr': 'ZeroDivisionError: ', 'level': 'error'}, {'regex': <_sre.SRE_Pattern object at 0x7f0cb37759d0>, 'level': 'critical'}, {'regex': <_sre.SRE_Pattern object at 0x7f0cb35f5b70>, 'level': 'critical'}, {'substr': 'ERROR - ', 'level': 'error'}], 'cwd': '/tmp/tmp9dxLR7', 'privileged': False}, attempt #1
[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - Running command: ['/usr/bin/python2.7', '-u', '/builds/task_1595246656/workspace/mozharness/external_tools/tooltool.py', 'fetch', '-m', u'/builds/task_1595246656/workspace/build/tests/raptor/raptor/tooltool-manifests/chrome-android/chrome80.manifest', '-o'] in /tmp/tmp9dxLR7
[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - Copy/paste: /usr/bin/python2.7 -u /builds/task_1595246656/workspace/mozharness/external_tools/tooltool.py fetch -m /builds/task_1595246656/workspace/build/tests/raptor/raptor/tooltool-manifests/chrome-android/chrome80.manifest -o
[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - Calling ['/usr/bin/python2.7', '-u', '/builds/task_1595246656/workspace/mozharness/external_tools/tooltool.py', 'fetch', '-m', u'/builds/task_1595246656/workspace/build/tests/raptor/raptor/tooltool-manifests/chrome-android/chrome80.manifest', '-o'] with output_timeout 600
[task 2020-07-20T12:13:21.829Z] 12:12:28 INFO - INFO - Attempting to fetch from 'http://localhost:8099/tooltool.mozilla-releng.net/'...
[task 2020-07-20T12:13:21.829Z] 12:13:20 INFO - INFO - File com.android.chrome-v.80.0.3987.162.apk fetched from http://localhost:8099/tooltool.mozilla-releng.net/ as /tmp/tmp9dxLR7/tmpNtYghG
[task 2020-07-20T12:13:21.829Z] 12:13:21 INFO - INFO - File integrity verified, renaming tmpNtYghG to com.android.chrome-v.80.0.3987.162.apk
[task 2020-07-20T12:13:21.829Z] 12:13:21 INFO - Return code: 0
[task 2020-07-20T12:13:21.829Z] 12:13:21 INFO - Running post-action listener: _resource_record_post_action
[task 2020-07-20T12:13:21.829Z] 12:13:21 INFO - [mozharness: 2020-07-20 12:13:21.805662Z] Finished install-chrome-android step (failed)
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - Uncaught exception: Traceback (most recent call last):
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - File "/builds/task_1595246656/workspace/mozharness/mozharness/base/script.py", line 2137, in run
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - self.run_action(action)
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - File "/builds/task_1595246656/workspace/mozharness/mozharness/base/script.py", line 2076, in run_action
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - self._possibly_run_method(method_name, error_if_missing=True)
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - File "/builds/task_1595246656/workspace/mozharness/mozharness/base/script.py", line 2031, in _possibly_run_method
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - return getattr(self, method_name)()
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - File "/builds/task_1595246656/workspace/mozharness/mozharness/mozilla/testing/raptor.py", line 483, in install_chrome_android
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - self.device.shell_output("settings put global verifier_verify_adb_installs 0")
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - File "/builds/task_1595246656/workspace/mozharness/mozharness/mozilla/testing/android.py", line 57, in device
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - adb = self.adb_path
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - File "/builds/task_1595246656/workspace/mozharness/mozharness/mozilla/testing/android.py", line 45, in adb_path
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - self.activate_virtualenv()
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - File "/builds/task_1595246656/workspace/mozharness/mozharness/base/python.py", line 442, in activate_virtualenv
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - exec(open(activate).read(), dict(file=activate))
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - IOError: [Errno 2] No such file or directory: '/builds/task_1595246656/workspace/build/venv/bin/activate_this.py'
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - Running post_fatal callback...
[task 2020-07-20T12:13:21.829Z] 12:13:21 FATAL - Exiting -1

Comment 1

[Greg Mierzwinski [:sparky]]

This is interesting, :agi (on mac) is also hitting this issue running a firefox test: https://matrix.to/#/!LfXZSWEroPFPMQcYmw:mozilla.org/$wWD5hPoLANc6cLMVVGKhcuiCn94B7bz3IhK-g4CRQ-k?via=mozilla.org&via=matrix.org&via=matrix.start9labs.com

Comment 2

[Greg Mierzwinski [:sparky]]

:bc, this seems to be the same error was triggered by your patch in bug 1486004: https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedTaskRun=SwX6y89nRfa98U1uEHwvWg.0&tier=1%2C2%2C3&revision=870dff289e73e612c99fd9ee9962fdca8d7288b8&searchStr=cold-youtube

Comment 3

[Greg Mierzwinski [:sparky]]

Actually it may have been regressed by bug 1408051.

Comment 4

[Bob Clary [:bc] (inactive)]

I'm clearing my NI since it doesn't appear this is due to bug 1486004.

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: Bug 1655164 - do not attempt to access device before virtualenv created.

Comment 6

[Bob Clary [:bc] (inactive)]

I had some improvement with this patch and the patch from D85581 on try and think this will improve the situation if not resolve it. The basic idea is to put the create-virtualenv action before any action that might access the device and thus require mozdevice be available.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5b4a3edffdfd
do not attempt to access device before virtualenv created. r=perftest-reviewers,sparky

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/5b4a3edffdfd

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:bc, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 10

[Bob Clary [:bc] (inactive)]

I think the change is simple enough and prevents potential errors if the chrome tests should ever be run on beta that it should be uplifted.

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

The tests don't run on beta.