1499902 - Intermittent AttributeError: 'NoneType' object has no attribute 'shell_output'

Status: RESOLVED FIXED

(Testing :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: bclary [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=206174268&repo=mozilla-central

https://queue.taskcluster.net/v1/task/buDwFTUERKGeFNVYUDQ-gA/runs/0/artifacts/public/logs/live_backing.log

Comment 1

[Bob Clary [:bc] (inactive)]

18:05:45     INFO - [mozharness: 2018-10-17 18:05:45.808919Z] Finished verify-device step (failed)
18:05:45    FATAL - Uncaught exception: Traceback (most recent call last):
18:05:45    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 2104, in run
18:05:45    FATAL -     self.run_action(action)
18:05:45    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 2043, in run_action
18:05:45    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
18:05:45    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/base/script.py", line 1983, in _possibly_run_method
18:05:45    FATAL -     return getattr(self, method_name)()
18:05:45    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 492, in verify_device
18:05:45    FATAL -     self.dump_perf_info()
18:05:45    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 257, in dump_perf_info
18:05:45    FATAL -     out = self.shell_output(cmd)
18:05:45    FATAL -   File "/builds/worker/workspace/mozharness/mozharness/mozilla/testing/android.py", line 336, in shell_output
18:05:45    FATAL -     return self.device.shell_output(cmd, timeout=30)
18:05:45    FATAL - AttributeError: 'NoneType' object has no attribute 'shell_output'
18:05:45    FATAL - Running post_fatal callback...

gbrown: Thoughts?

Comment 2

[Geoff Brown [:gbrown]]

self.device is a property which includes a silent try/catch. I think there's something wrong with the physical device and ADBAndroid() is throwing.

Comment 3

[Geoff Brown [:gbrown]]

That would explain why:

  INFO - New mozdevice with adb=..., device=...

is missing.

Comment 4

[Bob Clary [:bc] (inactive)]

But for this to happen, shell.device is None. Ah, I see. In android.py

    @property
    def device(self):
        if not self._device:
            try:
                import mozdevice
                self._device = mozdevice.ADBAndroid(adb=self.adb_path,
                                                    device=self.device_serial,
                                                    verbose=True)
                self.info("New mozdevice with adb=%s, device=%s" %
                          (self.adb_path, self.device_serial))
            except Exception:
                # As in adb_path, above.
                pass
        return self._device

We should change that except to match adb_path's where it catches AttributeError if that is the error we want to ignore.

Comment 5

[Geoff Brown [:gbrown]]

Attachment: ignore fewer exceptions in AndroidMixin device property

I think this is all that's needed to transform this bug into something more understandable.

Doesn't break normal stuff:

https://treeherder.mozilla.org/#/jobs?repo=try&tier=1%2C2%2C3&revision=50c249da9d1597330ac8079475ad4d44ed901310

Comment 6

[Bob Clary [:bc] (inactive)]

Comment on attachment 9018444 [details] [diff] [review]
ignore fewer exceptions in AndroidMixin device property

Review of attachment 9018444 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm. r+

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ab5f788c2759
Improve device failure handling in AndroidMixin; r=bc

Comment 8

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/ab5f788c2759

Comment 9

[Intermittent Failures Robot]

4 failures in 2759 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 2
* autoland: 1
* mozilla-central: 1

Platform breakdown:
* android-hw-p2-8-0-arm7-api-16: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1499902&startday=2018-10-15&endday=2018-10-21&tree=all