1230105 - Broken pipe errors when running tests locally in parallel

Status: NEW

(www.mozilla.org :: Bedrock, defect)

Description

[Alex Gibson [:agibson]]

When running some of our functional tests against a local bedrock instance in parallel (multiple tests run at once), it seems tests that rely on ajax responses [1] seem to fail, because running multiple requests interrupts the socket.

This doesn't happen when running against a deployed instance, such as dev, stage etc, only running locally. The tests pass fine locally when run in serial.

Is there anything we can do to mitigate this so we can run these tests locally in parallel?

This is the traceback:

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 270, in send_headers
    self.send_preamble()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 194, in send_preamble
    'Date: %s\r\n' % format_date_time(time.time())
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 328, in write
    self.flush()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
[03/Dec/2015 02:41:28] "GET /en-US/contact/spaces/mountain-view/ HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58296)
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/Users/agibson/Sites/bedrock/venv/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 129, in __init__
    super(WSGIRequestHandler, self).__init__(*args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 283, in close
    self.flush()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
[03/Dec/2015 02:41:28] "GET /en-US/contact/communities/north-america/ HTTP/1.1" 200 62284
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 128, in finish_response
    self.write(data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 212, in write
    self.send_headers()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/wsgiref/handlers.py", line 271, in send_headers
    self._write(str(self.headers))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 328, in write
    self.flush()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
[03/Dec/2015 02:41:28] "GET /en-US/contact/communities/north-america/ HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 58306)
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 599, in process_request_thread
    self.finish_request(request, client_address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/Users/agibson/Sites/bedrock/venv/lib/python2.7/site-packages/django/core/servers/basehttp.py", line 129, in __init__
    super(WSGIRequestHandler, self).__init__(*args, **kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 657, in __init__
    self.finish()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/SocketServer.py", line 716, in finish
    self.wfile.close()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 283, in close
    self.flush()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 307, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------


[1] https://github.com/mozilla/bedrock/pull/3627

Comment 1

[Alex Gibson [:agibson]]

Paul, any ideas here?

For now we're adding a note to the docs for devs to run tests locally in serial, but it would be nice if there was some way to mitigate this.

Comment 2

[Paul [:pmac] McLanahan]

Seems like it's just a case of the dev server not being able to handle parallel traffic. An alternative would be to run the site using a real server like gunicorn, but that would be a bit more setup. How are you running the server to test against now? Is it just pytest-django?

Comment 3

[Dave Hunt [:davehunt] [he/him] ⌚GMT]

> How are you running the server to test against now? Is it just pytest-django?

We're not using pytest-django at the moment. The server is started manually by the dev, presumably by following the existing documentation. At least that's what I've been doing.

Comment 4

[Paul [:pmac] McLanahan]

Just FYI then, pytest-django is very good. I use it for the redirect tests and it handles starting a server and using it. It seems to also handle using pytest-xdist pretty well. Not sure if it would solve your problem or not though.

Other option like I said above is to run it with gunicorn. This is how prod is run in our new AWS infra:

https://github.com/mozilla/bedrock/blob/master/docker/run-prod.sh#L12