Closed Bug 1331455 Opened 7 years ago Closed 7 years ago

Intermittent Android OSError: [Errno 2] No such file or directory - unable to run netstat

Categories

(Firefox for Android Graveyard :: Testing, defect, P1)

defect

Tracking

(firefox50 unaffected, firefox51 unaffected, firefox52 unaffected, firefox53 fixed)

RESOLVED FIXED
Firefox 53
Tracking Status
firefox50 --- unaffected
firefox51 --- unaffected
firefox52 --- unaffected
firefox53 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

+++ This bug was initially created as a clone of Bug #1324210 +++

Several Android intermittent failures are recorded in bug 1324210, beginning Jan 12, caused by a failure to run netstat. I think this was regressed by bug 1330408, which switched some Android jobs to a Ubuntu 16 docker image.
https://treeherder.mozilla.org/logviewer.html#?repo=autoland&job_id=69364108&lineNumber=1499

[task 2017-01-16T19:55:25.314223Z] 19:55:25     INFO - Running netstat -a -p -n -t -u
[task 2017-01-16T19:55:25.359580Z] 19:55:25     INFO - Running post-action listener: _resource_record_post_action
[task 2017-01-16T19:55:25.359922Z] 19:55:25     INFO - [mozharness: 2017-01-16 19:55:25.359765Z] Finished verify-emulator step (failed)
[task 2017-01-16T19:55:25.361500Z] 19:55:25    FATAL - Uncaught exception: Traceback (most recent call last):
[task 2017-01-16T19:55:25.361937Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/script.py", line 2028, in run
[task 2017-01-16T19:55:25.362362Z] 19:55:25    FATAL -     self.run_action(action)
[task 2017-01-16T19:55:25.362965Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/script.py", line 1967, in run_action
[task 2017-01-16T19:55:25.363220Z] 19:55:25    FATAL -     self._possibly_run_method(method_name, error_if_missing=True)
[task 2017-01-16T19:55:25.363662Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/mozharness/base/script.py", line 1907, in _possibly_run_method
[task 2017-01-16T19:55:25.364095Z] 19:55:25    FATAL -     return getattr(self, method_name)()
[task 2017-01-16T19:55:25.364539Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 627, in verify_emulator
[task 2017-01-16T19:55:25.364974Z] 19:55:25    FATAL -     emulator_ok = self._retry(max_restarts, 10, self._verify_emulator_and_restart_on_fail, "Check emulator")
[task 2017-01-16T19:55:25.365404Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 244, in _retry
[task 2017-01-16T19:55:25.365895Z] 19:55:25    FATAL -     status = func()
[task 2017-01-16T19:55:25.366309Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 339, in _verify_emulator_and_restart_on_fail
[task 2017-01-16T19:55:25.366701Z] 19:55:25    FATAL -     self._dump_host_state()
[task 2017-01-16T19:55:25.367231Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 376, in _dump_host_state
[task 2017-01-16T19:55:25.367550Z] 19:55:25    FATAL -     self._run_proc(['netstat', '-a', '-p', '-n', '-t', '-u'])
[task 2017-01-16T19:55:25.368049Z] 19:55:25    FATAL -   File "/home/worker/workspace/mozharness/scripts/android_emulator_unittest.py", line 253, in _run_proc
[task 2017-01-16T19:55:25.368440Z] 19:55:25    FATAL -     p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
[task 2017-01-16T19:55:25.368835Z] 19:55:25    FATAL -   File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
[task 2017-01-16T19:55:25.369327Z] 19:55:25    FATAL -     errread, errwrite)
[task 2017-01-16T19:55:25.369731Z] 19:55:25    FATAL -   File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
[task 2017-01-16T19:55:25.370176Z] 19:55:25    FATAL -     raise child_exception
[task 2017-01-16T19:55:25.370764Z] 19:55:25    FATAL - OSError: [Errno 2] No such file or directory
[task 2017-01-16T19:55:25.371276Z] 19:55:25    FATAL - Running post_fatal callback...
[task 2017-01-16T19:55:25.371776Z] 19:55:25    FATAL - Exiting -1
[task 2017-01-16T19:55:25.372383Z] 19:55:25     INFO - Running post-run listener: _resource_record_post_run
android_emulator_unittest.py normally does not call netstat. It only calls netstat when the emulator fails to start up. Normally, when that happens, mozharness dumps additional info - such as netstat output - and then retries the job automatically; those retries are now being reported as netstat failures. 

I guess netstat is simply missing from the Ubuntu 16 docker image.
Verified on a loaner that netstat is not installed, and can be:

root@taskcluster-worker:~# netstat
bash: netstat: command not found
root@taskcluster-worker:~# apt install net-tools
...
root@taskcluster-worker:~# netstat -a -p -n -t -u
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.17.0.5:44855        91.189.88.162:80        TIME_WAIT   -               
root@taskcluster-worker:~#
Add net-tools to ubuntu 16.04 docker image, for netstat command, used by Android test jobs.

No harm done: https://treeherder.mozilla.org/#/jobs?repo=try&revision=412b475b80a2a2879afd11ff87bb0feeb9d730da
Attachment #8827285 - Flags: review?(jmaher)
Priority: -- → P1
Attachment #8827285 - Flags: review?(jmaher) → review+
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0159194557cf
Add net-tools to Ubuntu 16 docker image, for netstat; r=jmaher
https://hg.mozilla.org/mozilla-central/rev/0159194557cf
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → Firefox 53
Product: Firefox for Android → Firefox for Android Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: