Closed Bug 1504757 Opened 7 years ago Closed 6 years ago

Intermittent raise ADBError("ADBDevice.__init__: ls could not be found")

Categories

(Testing :: Mozbase, defect, P5)

Product:
Testing
Component:
Mozbase
Type:
defect

Tracking

(firefox66 wontfix, firefox67 affected)

Status:
RESOLVED WORKSFORME
Tracking Flags:
Tracking Status
firefox66 --- wontfix
firefox67 --- affected

People

(Reporter: intermittent-bug-filer, Assigned: bc)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell infra])

android-em-7-0-x86 cases here are all from kwierso's try runs of wpt - curious. Those will be mitigated by bug 1506254, if I can ever make that work. I don't understand how this is happening at all, especially in cases like https://treeherder.mozilla.org/logviewer.html#?job_id=212740036&repo=try&lineNumber=954 [task 2018-11-19T23:18:02.544Z] 23:18:02 INFO - /system/bin/ls -1A supported ... [task 2018-11-19T23:18:16.410Z] 23:18:16 ERROR - File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb.py", line 655, in __init__ [task 2018-11-19T23:18:16.411Z] 23:18:16 ERROR - raise ADBError("ADBDevice.__init__: ls could not be found") [task 2018-11-19T23:18:16.411Z] 23:18:16 ERROR - ADBError: ADBDevice.__init__: ls could not be found
cbrindusan: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=218233447&repo=autoland&lineNumber=1176 is actually this bug not bug 1498037. See the full error message rather than just the first line of the exception where Traceback occurs.

This bug failed 35 times in the last 7 days. Occurs on android-hw-g5-7-0-arm7-api-16 and on android-hw-p2-8-0-arm7-api-16
on opt build types.

Recent log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=231011175&repo=mozilla-central&lineNumber=894

status-firefox66: --- → wontfix
status-firefox67: --- → affected
Priority: -- → P5

Most of the recent failures here are for raptor tests. A variety of errors are reported, most not matching this bug's description. "no raptor test results were found" seems common.

Whiteboard: [stockwell needswork:owner] → [stockwell infra]

There are 24 total failures all raptor, on android.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235770330&repo=mozilla-central&lineNumber=1184

https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=235770330&repo=mozilla-central&lineNumber=1184

00:42:00 INFO - raptor-main starting geckoview
00:42:00 INFO - adb shell_output: adb -s FA83W1A02560 wait-for-device shell am force-stop org.mozilla.geckoview_example; echo adb_returncode=$?, timeout: None, root: False, timedout: None, exitcode: 0, output:
00:42:00 INFO - adb Granting important runtime permissions to org.mozilla.geckoview_example

00:44:51 INFO - mozproxy Stopping mitmproxy playback, killing process 536
00:44:51 INFO - mozproxy Successfully killed the mitmproxy playback process
00:44:51 INFO - raptor-main removing webext /builds/worker/workspace/build/tests/raptor/raptor/../webext/raptor
00:44:51 INFO - results-handler summarizing raptor test results
00:44:51 INFO - raptor-output error: no raptor test results found for raptor-tp6m-google-maps-geckoview
00:44:51 ERROR - Traceback (most recent call last):
00:44:51 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 869, in <module>
00:44:51 INFO - main()
00:44:51 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 837, in main
00:44:51 INFO - success = raptor.process_results(raptor_test_names)
00:44:51 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/raptor.py", line 342, in process_results
00:44:51 INFO - return self.results_handler.summarize_and_output(self.config, test_names)
00:44:51 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/results.py", line 80, in summarize_and_output
00:44:51 INFO - output.combine_browser_cycles()
00:44:51 INFO - File "/builds/worker/workspace/build/tests/raptor/raptor/output.py", line 214, in combine_browser_cycles
00:44:51 INFO - for _index, suite in enumerate(self.summarized_results['suites']):
00:44:51 INFO - KeyError: 'suites'
00:44:52 ERROR - Return code: 1
00:44:52 WARNING - setting return code to 1
00:44:52 INFO - Killing logcat pid 445.
00:44:52 CRITICAL - PERFHERDER_DATA was seen 0 times, expected 1.
00:44:52 INFO - copying raptor results to upload dir:
00:44:52 INFO - /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
00:44:52 INFO - copying raptor results from /builds/worker/workspace/build/raptor.json to /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
00:44:52 CRITICAL - Error copying results /builds/worker/workspace/build/raptor.json to upload dir /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json
00:44:52 INFO - [Errno 2] No such file or directory: u'/builds/worker/workspace/build/raptor.json'

Incorrect classifications continue, as shown in comment 25. It is difficult to find ADBError in any of the failures reported against this bug now.

I think what is happening is that bug 1528070 is the only suggestion for the common error

Error copying results /builds/worker/workspace/build/raptor.json to upload dir /builds/worker/workspace/build/blobber_upload_dir/perfherder-data.json

and bug 1528070 was closed as a dup of 1504757 earlier (when there was an associated ADBError).

:rwood -- Can you investigate? It looks like "KeyError: 'suites'" is a common failure for some android raptor tests.

Flags: needinfo?(rwood)

(In reply to Geoff Brown [:gbrown] from comment #27)

:rwood -- Can you investigate? It looks like "KeyError: 'suites'" is a common failure for some android raptor tests.

Thanks, will do

Flags: needinfo?(rwood)

Ah yes I see the issue, Filed Bug 1538786 to address this and will make a patch ASAP.

Depends on: 1538786

Only 2 out of 11 failures during the last week were this failure. The others were all misclassifications of other errors.

Assignee: nobody → bob
Status: NEW → ASSIGNED
Component: Testing → Mozbase
Flags: needinfo?(aryx.bugmail)
Product: Firefox for Android → Testing
See Also: → 1517827

20 of these are the ls could not be found problem. The others are misclassifications. The ls problems are scattered among emulators and hardware. If we don't know about errors like the failures to download we can't investigate or fix them.

23:27:39     INFO -  mozdevice.adb.ADBProcessError: args: adb wait-for-device reverse tcp:44929 tcp:44929, exitcode: 1, stdout: adb: error: cannot bind listener: Address already in use
17:06:39     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563037473/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpr6zwM0.mozrunner/', '-L']' returned non-zero exit status 255
1
17:24:21     INFO -  KeyError: u'raptor-tp6m-imdb-geckoview-cold'
17:13:40     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563037897/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpJUhiEN.mozrunner/', '-L']' returned non-zero exit status 255
17:07:03     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563037488/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpy2xSKx.mozrunner/', '-L']' returned non-zero exit status 255
16:29:19     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563035227/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpqCZjOA.mozrunner/', '-L']' returned non-zero exit status 255
16:34:13     INFO -  mozproxy ERROR - The following files failed: 'mitmproxy-2.0.2-linux.tar.gz'
15:42:36     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563032433/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpwOdgIE.mozrunner/', '-L']' returned non-zero exit status 255
15:41:21     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563032326/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpuP4Z8Q.mozrunner/', '-L']' returned non-zero exit status 255
1
13:19:07     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563023830/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpzY8E5f.mozrunner/', '-L']' returned non-zero exit status 255
13:34:43     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563024725/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpwpDuAZ.mozrunner/', '-L']' returned non-zero exit status 255
13:05:02     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563022978/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpeP4UcI.mozrunner/', '-L']' returned non-zero exit status 255
07:45:06     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1563003751/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmp7loqqy.mozrunner/', '-L']' returned non-zero exit status 255
0
01:49:44     INFO -  mozproxy ERROR - The following files failed: 'host-utils-68.0a1.en-US.linux-x86_64.tar.gz'
01
22:12:14     INFO -  mozdevice.adb.ADBProcessError: args: adb wait-for-device reverse tcp:43445 tcp:43445, exitcode: 1, stdout: adb: error: cannot bind listener: Address already in use
[task 2019-07-12T20:45:27.159Z] 20:45:27    ERROR - No tests run or test summary not found
[20:56:04     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1562964840/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpEXYevW.mozrunner/', '-L']' returned non-zero exit status 255
20:16:20     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1562962428/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmp1dIYNi.mozrunner/', '-L']' returned non-zero exit status 255
9:24:15     INFO -  mozproxy ERROR - The following files failed: 'host-utils-68.0a1.en-US.linux-x86_64.tar.gz'
19:08:11     INFO -  subprocess.CalledProcessError: Command '['/builds/task_1562958371/workspace/testing/mozproxy/host-utils-68.0a1.en-US.linux-x86_64/certutil', '-d', 'sql:/tmp/tmpxW9zi2.mozrunner/', '-L']' returned non-zero exit status 255
17:11:07     INFO -  KeyError: u'raptor-tp6m-youtube-geckoview-cold'
[task 2019-07-11T02:05:41.760Z] 02:05:41    ERROR - No tests run or test summary not found
[task 2019-07-11T01:17:29.915Z] 01:17:29    ERROR - No tests run or test summary not found
[task 2019-07-10T22:41:03.356Z] 22:41:03    ERROR - No tests run or test summary not found
[task 2019-07-10T18:25:49.524Z] 18:25:49    ERROR - No tests run or test summary not found
11:01:04     INFO -  mozproxy INFO - ...failed to fetch 'mitmproxy-4.0.4-linux.tar.gz' from https://tooltool.mozilla-releng.net/
[task 2019-07-09T09:21:45.672Z] 09:21:45    ERROR - No tests run or test summary not found
[task 2019-07-09T04:50:28.724Z] 04:50:28     INFO -  Automation Error: No crash directory (/sdcard/tests/profile/minidumps) found on remote device
[task 2019-07-08T16:50:12.826Z] 16:50:12     INFO -  Automation Error: No crash directory (/sdcard/tests/profile/minidumps) found on remote device
[task 2019-07-08T16:19:54.146Z] 16:19:54     INFO -  Automation Error: No crash directory (/sdcard/tests/profile/minidumps) found on remote device
[task 2019-07-08T14:01:43.312Z] 14:01:43     INFO -  Automation Error: No crash directory (/sdcard/tests/profile/minidumps) found on remote device
[task 2019-07-08T12:39:58.245Z] 12:39:58    ERROR - No tests run or test summary not found
[task 2019-07-08T06:43:23.554Z] 06:43:23    ERROR - No tests run or test summary not found
See Also: → 1567332

suggest that android-em failures are temporary or intermittent: test retry 1 second later finds ls.

I can confirm: I just started running some command sequences in short loops and generally found success with <5 iterations.

There are 49 failures associated to this bug in the last 7 days. These are occurring on android-hw-g5-7-0-arm7-api-16, android-hw-p2-8-0-android-aarch64, android-hw-p2-8-0-arm7-api-16 and android-em-7-0-x86_64 all build types.

The autoland failure is misclassified. It should have been ERROR - The following files failed: 'mitmproxy-4.0.4-linux.tar.gz'.

(In reply to Intermittent Failures Robot from comment #69)

3 failures in 4530 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

These were all misclassifications.

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.

Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE

New occurrence: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=272543016&repo=autoland&lineNumber=1098

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
See Also: → 1605771

(In reply to Intermittent Failures Robot from comment #83)

2 failures in 3773 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

These were misclassified: TEST-UNEXPECTED-FAIL: no raptor test results were found for raptor-unity-webgl-firefox

Considering this is an Android only error message and None of the platforms are Android, I can say without further ado that these are all misclassified.

Sorry about that, we'll try to avoid that in the future.

The missclassified failures have been updated.

I'm going to call this wfm and call it a day. There have been no instances of this actual error going back to at least December 2019. The others have all been misclassifications.

Do not reopen this bug. If this occurs again, let's start a new one with a clean history.

Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → WORKSFORME

All recent classifications are still wrong.

You need to log in before you can comment on or make changes to this bug.