Intermittent Android OSError: [Errno 2] No such file or directory - unable to run netstat

RESOLVED FIXED in Firefox 53



Firefox for Android
a year ago
a year ago


(Reporter: gbrown, Assigned: gbrown)



Firefox 53
Dependency tree / graph

Firefox Tracking Flags

(firefox50 unaffected, firefox51 unaffected, firefox52 unaffected, firefox53 fixed)



(1 attachment)



a year ago
+++ This bug was initially created as a clone of Bug #1324210 +++

Several Android intermittent failures are recorded in bug 1324210, beginning Jan 12, caused by a failure to run netstat. I think this was regressed by bug 1330408, which switched some Android jobs to a Ubuntu 16 docker image.

Comment 1

a year ago

[task 2017-01-16T19:55:25.314223Z] 19:55:25     INFO - Running netstat -a -p -n -t -u
[task 2017-01-16T19:55:25.359580Z] 19:55:25     INFO - Running post-action listener: _resource_record_post_action
[task 2017-01-16T19:55:25.359922Z] 19:55:25     INFO - [mozharness: 2017-01-16 19:55:25.359765Z] Finished verify-emulator step (failed)
[task 2017-01-16T19:55:25.361500Z] 19:55:25    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2017-01-16T19:55:25.361937Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/", line 2028, in run
[task 2017-01-16T19:55:25.362362Z] 19:55:25    FATAL -     self.run_action(action)
[task 2017-01-16T19:55:25.362965Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/", line 1967, in run_action
[task 2017-01-16T19:55:25.363220Z] 19:55:25    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2017-01-16T19:55:25.363662Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/", line 1907, in _possibly_run_method
[task 2017-01-16T19:55:25.364095Z] 19:55:25    FATAL -     return getattr(self, method_name)()
[task 2017-01-16T19:55:25.364539Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/", line 627, in verify_emulator
[task 2017-01-16T19:55:25.364974Z] 19:55:25    FATAL -     emulator_ok = self._retry(max_restarts, 10, self._verify_emulator_and_restart_on_fail, "Check emulator")
[task 2017-01-16T19:55:25.365404Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/", line 244, in _retry
[task 2017-01-16T19:55:25.365895Z] 19:55:25    FATAL -     status = func()
[task 2017-01-16T19:55:25.366309Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/", line 339, in _verify_emulator_and_restart_on_fail
[task 2017-01-16T19:55:25.366701Z] 19:55:25    FATAL -     self._dump_host_state()
[task 2017-01-16T19:55:25.367231Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/", line 376, in _dump_host_state
[task 2017-01-16T19:55:25.367550Z] 19:55:25    FATAL -     self._run_proc(['netstat', '-a', '-p', '-n', '-t', '-u'])
[task 2017-01-16T19:55:25.368049Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/", line 253, in _run_proc
[task 2017-01-16T19:55:25.368440Z] 19:55:25    FATAL -     p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
[task 2017-01-16T19:55:25.368835Z] 19:55:25    FATAL -   File "/usr/lib/python2.7/", line 711, in __init__
[task 2017-01-16T19:55:25.369327Z] 19:55:25    FATAL -     errread, errwrite)
[task 2017-01-16T19:55:25.369731Z] 19:55:25    FATAL -   File "/usr/lib/python2.7/", line 1343, in _execute_child
[task 2017-01-16T19:55:25.370176Z] 19:55:25    FATAL -     raise child_exception
[task 2017-01-16T19:55:25.370764Z] 19:55:25    FATAL - OSError: [Errno 2] No such file or directory
[task 2017-01-16T19:55:25.371276Z] 19:55:25    FATAL - Running post_fatal callback...
[task 2017-01-16T19:55:25.371776Z] 19:55:25    FATAL - Exiting -1
[task 2017-01-16T19:55:25.372383Z] 19:55:25     INFO - Running post-run listener: _resource_record_post_run

Comment 2

a year ago normally does not call netstat. It only calls netstat when the emulator fails to start up. Normally, when that happens, mozharness dumps additional info - such as netstat output - and then retries the job automatically; those retries are now being reported as netstat failures. 

I guess netstat is simply missing from the Ubuntu 16 docker image.

Comment 3

a year ago
Verified on a loaner that netstat is not installed, and can be:

root@taskcluster-worker:~# netstat
bash: netstat: command not found
root@taskcluster-worker:~# apt install net-tools
root@taskcluster-worker:~# netstat -a -p -n -t -u
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0        TIME_WAIT   -               

Comment 4

a year ago
Created attachment 8827285 [details] [diff] [review]
add net-tools to ubuntu 16, for netstat

Add net-tools to ubuntu 16.04 docker image, for netstat command, used by Android test jobs.

No harm done:
Attachment #8827285 - Flags: review?(jmaher)


a year ago
Priority: -- → P1
Attachment #8827285 - Flags: review?(jmaher) → review+

Comment 5

a year ago
Pushed by
Add net-tools to Ubuntu 16 docker image, for netstat; r=jmaher

Comment 6

a year ago
Last Resolved: a year ago
status-firefox53: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 53
status-firefox50: --- → unaffected
status-firefox51: --- → unaffected
status-firefox52: --- → unaffected

Comment 7

a year ago
10 failures in 690 pushes (0.014 failures/push) were associated with this bug in the last 7 days.  

Repository breakdown:
* mozilla-inbound: 6
* autoland: 3
* mozilla-central: 1

Platform breakdown:
* android-4-3-armv7-api15: 10

For more details, see:
You need to log in before you can comment on or make changes to this bug.