1331455 - Intermittent Android OSError: [Errno 2] No such file or directory - unable to run netstat

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect, P1)

Description

[Geoff Brown [:gbrown]]

+++ This bug was initially created as a clone of Bug #1324210 +++

Several Android intermittent failures are recorded in bug 1324210, beginning Jan 12, caused by a failure to run netstat. I think this was regressed by bug 1330408, which switched some Android jobs to a Ubuntu 16 docker image.

Comment 1

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=69364108&lineNumber=1499

[task 2017-01-16T19:55:25.314223Z] 19:55:25     INFO - Running netstat -a -p -n -t -u
[task 2017-01-16T19:55:25.359580Z] 19:55:25     INFO - Running post-action listener: _resource_record_post_action
[task 2017-01-16T19:55:25.359922Z] 19:55:25     INFO - [mozharness: 2017-01-16 19:55:25.359765Z] Finished verify-emulator step (failed)
[task 2017-01-16T19:55:25.361500Z] 19:55:25    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2017-01-16T19:55:25.361937Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/script.py", line 2028, in run
[task 2017-01-16T19:55:25.362362Z] 19:55:25    FATAL -     self.run_action(action)
[task 2017-01-16T19:55:25.362965Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/script.py", line 1967, in run_action
[task 2017-01-16T19:55:25.363220Z] 19:55:25    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2017-01-16T19:55:25.363662Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/script.py", line 1907, in _possibly_run_method
[task 2017-01-16T19:55:25.364095Z] 19:55:25    FATAL -     return getattr(self, method_name)()
[task 2017-01-16T19:55:25.364539Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 627, in verify_emulator
[task 2017-01-16T19:55:25.364974Z] 19:55:25    FATAL -     emulator_ok = self._retry(max_restarts, 10, self._verify_emulator_and_restart_on_fail, "Check emulator")
[task 2017-01-16T19:55:25.365404Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 244, in _retry
[task 2017-01-16T19:55:25.365895Z] 19:55:25    FATAL -     status = func()
[task 2017-01-16T19:55:25.366309Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 339, in _verify_emulator_and_restart_on_fail
[task 2017-01-16T19:55:25.366701Z] 19:55:25    FATAL -     self._dump_host_state()
[task 2017-01-16T19:55:25.367231Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 376, in _dump_host_state
[task 2017-01-16T19:55:25.367550Z] 19:55:25    FATAL -     self._run_proc(['netstat', '-a', '-p', '-n', '-t', '-u'])
[task 2017-01-16T19:55:25.368049Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 253, in _run_proc
[task 2017-01-16T19:55:25.368440Z] 19:55:25    FATAL -     p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
[task 2017-01-16T19:55:25.368835Z] 19:55:25    FATAL -   File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
[task 2017-01-16T19:55:25.369327Z] 19:55:25    FATAL -     errread, errwrite)
[task 2017-01-16T19:55:25.369731Z] 19:55:25    FATAL -   File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
[task 2017-01-16T19:55:25.370176Z] 19:55:25    FATAL -     raise child_exception
[task 2017-01-16T19:55:25.370764Z] 19:55:25    FATAL - OSError: [Errno 2] No such file or directory
[task 2017-01-16T19:55:25.371276Z] 19:55:25    FATAL - Running post_fatal callback...
[task 2017-01-16T19:55:25.371776Z] 19:55:25    FATAL - Exiting -1
[task 2017-01-16T19:55:25.372383Z] 19:55:25     INFO - Running post-run listener: _resource_record_post_run

Comment 2

[Geoff Brown [:gbrown]]

android_emulator_unittest.py normally does not call netstat. It only calls netstat when the emulator fails to start up. Normally, when that happens, mozharness dumps additional info - such as netstat output - and then retries the job automatically; those retries are now being reported as netstat failures. 

I guess netstat is simply missing from the Ubuntu 16 docker image.

Comment 3

[Geoff Brown [:gbrown]]

Verified on a loaner that netstat is not installed, and can be:

root@taskcluster-worker:~# netstat
bash: netstat: command not found
root@taskcluster-worker:~# apt install net-tools
...
root@taskcluster-worker:~# netstat -a -p -n -t -u
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.17.0.5:44855        91.189.88.162:80        TIME_WAIT   -               
root@taskcluster-worker:~#

Comment 4

[Geoff Brown [:gbrown]]

Attachment: add net-tools to ubuntu 16, for netstat

Add net-tools to ubuntu 16.04 docker image, for netstat command, used by Android test jobs.

No harm done: https://treeherder.mozilla.org/#/jobs?repo=try&revision=412b475b80a2a2879afd11ff87bb0feeb9d730da

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0159194557cf
Add net-tools to Ubuntu 16 docker image, for netstat; r=jmaher

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/0159194557cf

Comment 7

[Intermittent Failures Robot]

10 failures in 690 pushes (0.014 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 6
* autoland: 3
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 10

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1331455&startday=2017-01-16&endday=2017-01-22&tree=all