1526034 - High Frequency OSError: [Errno 24] Too many open files on Raptor failures

Status: RESOLVED INCOMPLETE

(Testing :: Raptor, defect, P5)

Description

[Treeherder Bug Filer]

#[markdown(off)]
Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=226901641&repo=autoland

https://queue.taskcluster.net/v1/task/eGa3C6tiQYy0Xx8eBLPZYw/runs/0/artifacts/public/logs/live_backing.log

10:06:29 ERROR - PID 1757 | JavaScript error: https://prebid.districtm.ca/lib.js, line 1: TypeError: s.default.selectById(...) is undefined
10:06:35 INFO - 127.0.0.1 - - [07/Feb/2019 10:06:35] "POST / HTTP/1.1" 200 -
10:06:35 INFO - raptor-control-server received webext_status: results received
10:06:35 INFO - 127.0.0.1 - - [07/Feb/2019 10:06:35] "POST / HTTP/1.1" 200 -
10:06:35 INFO - raptor-control-server received webext_results: {u'subtest_unit': u'ms', u'name': u'raptor-tp6-imgur-firefox', u'type': u'pageload', u'measurements': {u'dcf': [1720, 732, 803, 1265, 574, 1198, 1471, 665, 1468, 1198, 722, 1501, 819, 598, 1042, 811, 511, 1483, 547, 487, 692, 573, 499, 679, 983], u'fcp': [633, 754, 822, 1292, 594, 628, 978, 446, 855, 581, 744, 863, 844, 222, 1068, 837, 532, 870, 566, 506, 715, 194, 514, 184, 570], u'fnbpaint': [632, 754, 822, 1292, 594, 628, 978, 446, 855, 581, 744, 863, 844, 222, 1068, 837, 532, 870, 566, 506, 715, 194, 514, 184, 162], u'ttfi': [5333, 19475, 3074, 16120, 14481, 2761, 12269, 2666, 1445, 4503, 2802, 3063, 15328, 2615, 3281, 12280, 2433, 4366, 2466, 6611, 1205, 2710, 9832, 2666, 2549]}, u'subtest_lower_is_better': True, u'browser': u'Firefox 67.0a1 20190207080753', u'lower_is_better': True, u'page': u'https://imgur.com/gallery/m5tYJL6', u'unit': u'ms', u'alert_threshold': 2}
10:06:35 INFO - results-handler received results in RaptorResultsHandler.add
10:06:35 INFO - 127.0.0.1 - - [07/Feb/2019 10:06:35] "POST / HTTP/1.1" 200 -
10:06:35 INFO - raptor-control-server received webext_status: __raptor_shutdownBrowser
10:06:35 INFO - raptor-control-server shutting down browser (pid: 1757)
10:06:35 INFO - 127.0.0.1 - - [07/Feb/2019 10:06:35] "POST / HTTP/1.1" 200 -
10:06:35 INFO - raptor-control-server received webext_status: Removed tab 1
10:06:35 INFO - PID 1757 | JavaScript error: https://secure.quantserve.com/quant.js, line 19: NS_ERROR_UNEXPECTED:
10:06:52 INFO - raptor-mitmproxy Stopping mitmproxy playback, klling process 1755
10:06:52 INFO - Exception in thread Thread-2:
10:06:52 ERROR - Traceback (most recent call last):
10:06:52 INFO - File "threading.py", line 914, in _bootstrap_inner
10:06:52 INFO - File "site-packages/watchdog/observers/api.py", line 146, in run
10:06:52 INFO - File "site-packages/watchdog/observers/polling.py", line 91, in queue_events
10:06:52 INFO - File "site-packages/watchdog/observers/polling.py", line 74, in <lambda>
10:06:52 INFO - File "site-packages/watchdog/utils/dirsnapshot.py", line 236, in __init__
10:06:52 INFO - File "site-packages/watchdog/utils/dirsnapshot.py", line 213, in walk
10:06:52 INFO - OSError: [Errno 24] Too many open files: '/Users/cltbld/tasks/task_1549562044/build/tests/raptor/raptor/playback'
10:06:52 INFO - Exception in thread ServerThread (0.0.0.0:8080):
10:06:52 ERROR - Traceback (most recent call last):
10:06:52 INFO - File "threading.py", line 914, in _bootstrap_inner
10:06:52 INFO - File "mitmproxy/master.py", line 32, in run
10:06:52 INFO - File "mitmproxy/net/tcp.py", line 951, in serve_forever
10:06:52 INFO - File "socket.py", line 195, in accept
10:06:52 INFO - OSError: [Errno 24] Too many open files
10:06:52 INFO - Replaying from files: ['/Users/cltbld/tasks/task_1549562044/testing/raptor/imgur.mp']
10:07:02 INFO - raptor-mitmproxy Successfully killed the mitmproxy playback process
10:07:02 INFO - raptor-mitmproxy Turning off the browser proxy

Comment 1

[Intermittent Failures Robot]

161 failures in 486 pushes (0.331 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 48
• try: 1
• mozilla-central: 2
• mozilla-inbound: 97
• mozilla-beta: 13

Platform breakdown:

• linux64: 68
• windows7-32: 2
• linux64-qr: 5
• windows10-64: 1
• macosx64-nightly: 1
• osx-10-10: 80
• windows10-64-ux: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1526034&startday=2019-02-07&endday=2019-02-07&tree=all

Comment 2

[Intermittent Failures Robot]

190 failures in 2658 pushes (0.071 failures/push) were associated with this bug in the last 7 days.

This is the #6 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 48
• try: 1
• mozilla-inbound: 126
• mozilla-central: 2
• mozilla-beta: 13

Platform breakdown:

• windows10-64: 1
• linux64: 86
• linux64-qr: 5
• windows7-32: 2
• windows10-64-ux: 4
• macosx64-nightly: 1
• osx-10-10: 91

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1526034&startday=2019-02-04&endday=2019-02-10&tree=all

Comment 3

[Intermittent Failures Robot]

6 failures in 2832 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 2
• mozilla-inbound: 2
• try: 1
• mozilla-central: 1

Platform breakdown:

• macosx64-nightly: 2
• osx-10-10: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1526034&startday=2019-02-11&endday=2019-02-17&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 2602 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 1

Platform breakdown:

• osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1526034&startday=2019-02-18&endday=2019-02-24&tree=all

Comment 5

[Intermittent Failures Robot]

1 failures in 2981 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• osx-10-10: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1526034&startday=2019-02-25&endday=2019-03-03&tree=all

Comment 6

[Firefox Bug Husbandry Bot]

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup