1512264 - WPT WebDriver client does not reuse HTTP connection, causing us to exhaust the ephemeral port range on Windows

Status: RESOLVED FIXED

(Testing :: geckodriver, defect, P1)

Description

[Andreas Tolfsen ❲:ato❳]

Although the wdspec WebDriver test client specifies the Keep-Alive
header instructing the HTTPD to make the connection persistent, we
don’t actually make use of this as the httplib.HTTPConnection object
is destroyed after each HTTP request attempt.

This causes us to use a lot of client TCP sockets on Windows and
with bug 1511388 we noticed the following in the log:

> [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

I ran into this problem once in the past but regrettably didn’t
file a bug for it.  I can’t pinpoint exactly what in smaug’s patch
(https://hg.mozilla.org/try/rev/8776fc15b7cd89e19224161a90dd87fa0f723fa5)
caused us to exhaust the available ports specifically on Windows.

We do however know that Windows has a significantly limited ephemeral
port range compared to other systems and that it doesn’t take much
to exhaust it.  It is also likely that httplib has a bug (maybe a
WinSock bug?) that it doesn’t properly close client sockets after
httplib.HTTPConnection is destroyed/goes out of scope.

More generally we want to persist the HTTP connection between calls
to the remote end HTTPD, so fixing this will likely yield somewhat
better performance also on other systems.

I have verified through a try run that the patch fixes the TVw
problem on Windows 10 x86 opt:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a67d6140506d1ee09782df142f71e7bccd2acf5&group_state=expanded

Comment 1

[Andreas Tolfsen ❲:ato❳]

Attachment: bug 1512264: webdriver: reuse http connection in test client;

Whilst we do set the Keep-Alive HTTP header to request the HTTPD to
use a persistent connection, we throw away the httplib.HTTPConnection
object after each request.  This will causes the HTTPD to serve us
a persistent connection, only to throw it away after the first request.

When the connection timeout is reached, httplib closes the connection
for us.  Unfortunately httplib does not reconnect for the next
request, so we need to manually check if the underlying socket
connection to the server is alive.

We check whether the connection has dropped or if it has unread,
buffered data using select() as a poller.  We could use poll() here
on available systems, which might give a theoretical performance
improvement since it gives O(number of file descriptors), but since
we only deal with one connection select() will probably be good enough.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ac21c2e0b040
webdriver: reuse http connection in test client; r=jgraham

Comment 3

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/14439 for changes under testing/web-platform/tests

Comment 4

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/14439
* continuous-integration/travis-ci/pr (https://travis-ci.org/web-platform-tests/wpt/builds/466147283?utm_source=github_status&utm_medium=notification)

Comment 5

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/ac21c2e0b040

Comment 6

[Web Platform Test Sync Bot (Matrix: #interop:mozilla.org)]

Upstream PR merged