[mozproxy] mitmproxy causes "OSError: [Errno 24] Too many open files"
Categories
(Testing :: Mozbase, defect)
Tracking
(Not tracked)
People
(Reporter: whimboo, Unassigned)
Details
Running the tp6 AWSY job on my machine I regularly see the following error:
mozproxy ERROR Exception in thread ServerThread (('::ffff:127.0.0.1', 61169, 0, 0)):
mozproxy ERROR Traceback (most recent call last):
mozproxy ERROR File "threading.py", line 916, in _bootstrap_inner
mozproxy ERROR File "mitmproxy/master.py", line 37, in run
mozproxy ERROR File "mitmproxy/net/tcp.py", line 607, in serve_forever
mozproxy ERROR File "socket.py", line 205, in accept
mozproxy ERROR OSError: [Errno 24] Too many open files
mozproxy ERROR
As it looks like mitmproxy's serve_forever method might leak file handles.
|
||
Comment 1•4 years ago
|
||
Have you seen this problem for a while, or did it just start recently? Bug 1697048 recently switched 'mach awsy-test' from py-2 to py-3; I wonder if that might be related?
|
Reporter | |
Comment 2•4 years ago
|
||
Interesting question, which I actually cannot answer. I didn't try older versions yet, but I could perfectly do now given that it reproduces all the time when I run the tp6 test suite. Previously I also saw that for wptserve (https://github.com/web-platform-tests/wpt/issues/27072).
|
|
Reporter | |
Comment 3•4 years ago
|
||
I just tried with Python 2.7.17 and it's the same behavior. So it's not related to the switch to Python 3.
|
|
Reporter | |
Comment 4•4 years ago
|
||
The problem actually always appears after the first set of tabs have been closed and the first page load for https://www.fandom.com takes place:
1615489814132 Marionette DEBUG 2 -> [0,469,"WebDriver:Navigate",{"url":"https://www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase"}]
1615489814134 Marionette TRACE [17] MarionetteCommands actor created for window id 2147483681
1615489814135 Marionette TRACE Received event beforeunload for about:blank
1615489845896 Marionette TRACE Received event beforeunload for about:blank
1615489846053 Marionette TRACE Received event pagehide for about:blank
1615489846807 Marionette TRACE [17] MarionetteEvents actor created for window id 2147483682
1615489846848 Marionette TRACE Received event DOMContentLoaded for about:neterror?e=netTimeout&u=https%3A//www.fandom.com/articles/fallout-76-will-live-and-die-on-the-creativity-of-its-playerbase&c=UTF-8&d=The%20server%20at%20www.fandom.com%20is%20taking%20too%20long%20to%20respond.
1615489846848 Marionette DEBUG 2 <- [1,469,{"error":"unknown error","message":"Reached error page: about:neterror?e=netTimeout&u=https%3A//www.fandom.com/article ... modules/EventEmitter.jsm:160:20\nreceiveMessage@chrome://marionette/content/actors/MarionetteEventsParent.jsm:40:25\n"},null]
The page cannot be loaded, and finally ends-up in a network timeout.
Greg, should those long page loads or network timeouts happen at all with mitmproxy serving the page? Maybe it's caused by the too many files open issue and cannot serve any new data?
|
|
Reporter | |
Comment 5•4 years ago
|
||
Interestingly the soft limit for file descriptors is kinda low on my MacOS Big Sur system. Not sure if that was similar on older releases. But if mitmproxy keeps a lot of files open, the limit will indeed be reached kinda quickly.
➜ ulimit -aS (soft limit)
-n: file descriptors 256
➜ ulimit -aH (hard limit)
-n: file descriptors unlimited
|
||
Comment 6•4 years ago
|
||
Long page loads or timeouts can definitely happen if something changed in Firefox or mitmproxy is having issues serving the data (corrupted/outdated recordings can also cause this). It sounds like having too many files open would be the cause of it here based on your last comment.
| Comment hidden (Intermittent Failures Robot) |
| Comment hidden (Intermittent Failures Robot) |
|
|
||
Updated•3 years ago
|
Description
•