Closed Bug 1503590 Opened 6 years ago Closed 6 years ago

Intermittent Android File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb_android.py", line 100, in __init__

Categories

(Firefox for Android Graveyard :: Testing, defect, P5)

Product:
Firefox for Android Graveyard
Component:
Testing
Type:
defect

Tracking

(firefox65 fixed)

Status:
RESOLVED FIXED
Milestone:
Firefox 65
Tracking Flags:
Tracking Status
firefox65 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: gbrown)

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

translate ADBError and ValueError to simple failure in is_boot_completed
1014 bytes, patch
bc
: review+
Details | Diff | Splinter Review 
Filed by: csabou [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=208899530&repo=autoland

https://queue.taskcluster.net/v1/task/NIPx0q5ST-KxBOoQ1g3pQQ/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/NIPx0q5ST-KxBOoQ1g3pQQ/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2018-10-31T14:36:22.950Z] 14:36:22     INFO - Running post-action listener: _resource_record_post_action
[task 2018-10-31T14:36:22.950Z] 14:36:22     INFO - Running post-action listener: _start_resource_monitoring
[task 2018-10-31T14:36:22.967Z] 14:36:22     INFO - Starting resource monitoring.
[task 2018-10-31T14:36:22.972Z] 14:36:22     INFO - [mozharness: 2018-10-31 14:36:22.971622Z] Finished create-virtualenv step (success)
[task 2018-10-31T14:36:22.972Z] 14:36:22     INFO - [mozharness: 2018-10-31 14:36:22.972088Z] Running verify-device step.
[task 2018-10-31T14:36:22.972Z] 14:36:22     INFO - Running pre-action listener: _resource_record_pre_action
[task 2018-10-31T14:36:22.972Z] 14:36:22     INFO - Running main action method: verify_device
[task 2018-10-31T14:36:22.972Z] 14:36:22     INFO - >> Check emulator: Attempt #1 of 5
[task 2018-10-31T14:36:22.972Z] 14:36:22     INFO - >> Verify Android boot completed: Attempt #1 of 30
[task 2018-10-31T14:36:22.987Z] 14:36:22     INFO - Using adb 1.0.40
[task 2018-10-31T14:36:26.205Z] 14:36:26     INFO - adbd running as root
[task 2018-10-31T14:36:26.205Z] 14:36:26     INFO - adbd restarted as root
[task 2018-10-31T14:36:26.412Z] 14:36:26     INFO - su 0 supported
[task 2018-10-31T14:36:26.621Z] 14:36:26     INFO - /system/bin/ls -1A supported
[task 2018-10-31T14:36:26.725Z] 14:36:26     INFO - Native cp support: True
[task 2018-10-31T14:36:26.828Z] 14:36:26     INFO - Native chmod -R support: True
[task 2018-10-31T14:36:26.932Z] 14:36:26     INFO - Native chown -R support: True
[task 2018-10-31T14:36:27.036Z] 14:36:27     INFO - Setting SELinux Permissive
[task 2018-10-31T14:36:27.244Z] 14:36:27     INFO - Running post-action listener: _resource_record_post_action
[task 2018-10-31T14:36:27.244Z] 14:36:27     INFO - [mozharness: 2018-10-31 14:36:27.244483Z] Finished verify-device step (failed)
[task 2018-10-31T14:36:27.245Z] 14:36:27    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2018-10-31T14:36:27.245Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 2089, in run
[task 2018-10-31T14:36:27.245Z] 14:36:27    FATAL -     self.run_action(action)
[task 2018-10-31T14:36:27.245Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 2028, in run_action
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 1983, in _possibly_run_method
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -     return getattr(self, method_name)()
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 489, in verify_device
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -     "Check emulator")
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 232, in _retry
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -     status = func()
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 196, in _verify_emulator_and_restart_on_fail
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -     emulator_ok = self._verify_emulator()
[task 2018-10-31T14:36:27.246Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 189, in _verify_emulator
[task 2018-10-31T14:36:27.247Z] 14:36:27    FATAL -     max_time=330)
[task 2018-10-31T14:36:27.247Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 232, in _retry
[task 2018-10-31T14:36:27.247Z] 14:36:27    FATAL -     status = func()
[task 2018-10-31T14:36:27.247Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 331, in is_boot_completed
[task 2018-10-31T14:36:27.247Z] 14:36:27    FATAL -     out = self.device.get_prop('sys.boot_completed', timeout=30)
[task 2018-10-31T14:36:27.247Z] 14:36:27    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 64, in device
[task 2018-10-31T14:36:27.247Z] 14:36:27    FATAL -     verbose=True)
[task 2018-10-31T14:36:27.247Z] 14:36:27    FATAL -   File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb_android.py", line 100, in __init__
[task 2018-10-31T14:36:27.248Z] 14:36:27    FATAL -     timeout=timeout))
[task 2018-10-31T14:36:27.248Z] 14:36:27    FATAL - ValueError: invalid literal for int() with base 10: ''
[task 2018-10-31T14:36:27.248Z] 14:36:27    FATAL - Running post_fatal callback...
[task 2018-10-31T14:36:27.248Z] 14:36:27    FATAL - Exiting -1
[task 2018-10-31T14:36:27.248Z] 14:36:27     INFO - Running post-run listener: _resource_record_post_run
[task 2018-10-31T14:36:27.292Z] 14:36:27     INFO - Total resource usage - Wall time: 4s; CPU: 37.0%; Read bytes: 0; Write bytes: 2401280; Read time: 0; Write time: 0
[task 2018-10-31T14:36:27.292Z] 14:36:27     INFO - TinderboxPrint: CPU usage<br/>37.3%
[task 2018-10-31T14:36:27.292Z] 14:36:27     INFO - TinderboxPrint: I/O read bytes / time<br/>0 / 0
[task 2018-10-31T14:36:27.292Z] 14:36:27     INFO - TinderboxPrint: I/O write bytes / time<br/>2,401,280 / 0
[task 2018-10-31T14:36:27.293Z] 14:36:27     INFO - TinderboxPrint: CPU guest<br/>2.3 (6.6%)
[task 2018-10-31T14:36:27.293Z] 14:36:27     INFO - TinderboxPrint: CPU idle<br/>20.1 (58.5%)
[task 2018-10-31T14:36:27.293Z] 14:36:27     INFO - TinderboxPrint: CPU system<br/>1.2 (3.6%)
[task 2018-10-31T14:36:27.293Z] 14:36:27     INFO - TinderboxPrint: CPU user<br/>10.7 (31.1%)
[task 2018-10-31T14:36:27.293Z] 14:36:27     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2018-10-31T14:36:27.293Z] 14:36:27     INFO - verify-device - Wall time: 4s; CPU: 37.0%; Read bytes: 0; Write bytes: 2401280; Read time: 0; Write time: 0
Component: Applications: MozharnessCore → Testing
Product: Release Engineering → Firefox for Android
QA Contact: jlund
Summary: Intermittent Android Uncaught exception: Traceback (most recent call last): after Finished verify-device step (failed) → Intermittent Android File "/builds/worker/workspace/build/venv/lib/python2.7/site-packages/mozdevice/adb_android.py", line 100, in __init__
ValueError: invalid literal for int() with base 10: '' typically means we haven't finished booting for some reason. Though the mozharness/mozharness/mozilla/testing/android.py attempts to retry via the _retry method it does not catch exceptions when it calls func(). If it caught ADBError and ValueError and kept trying for the maximum attempts it might help.

gbrown?
Flags: needinfo?(gbrown)
Sounds good -- will give it a try.
Assignee: nobody → gbrown
Flags: needinfo?(gbrown)
Comment on attachment 9028026 [details] [diff] [review]
translate ADBError and ValueError to simple failure in is_boot_completed

Review of attachment 9028026 [details] [diff] [review]:
-----------------------------------------------------------------

r+ we can try it and see if it helps.
Attachment #9028026 - Flags: review?(bob) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c0caa96ca64a
Improve resilience when initializing mozdevice from mozharness; r=bc
https://hg.mozilla.org/mozilla-central/rev/c0caa96ca64a
Status: NEW → RESOLVED
Closed: 6 years ago
status-firefox65: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 65
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: