1386816 - Kill ssltunnel on port 8888 if we have started it previously and it's orphaned

Status: RESOLVED FIXED

(Testing :: Mochitest, enhancement)

Description

[(no longer active)]

See the example scenario below:

$ ./mach mochitest editor/libeditor/tests/test_bug439808.html --debugger=rr

######
### Now running mochitest-plain.
######

Checking for orphan ssltunnel processes...
ehsan    13937  2757  0 16:45 pts/3    00:00:00 /home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin/ssltunnel /tmp/ssltunnelWyz8Po.cfg
Checking for orphan xpcshell processes...
ehsan    13926  2757  0 16:45 pts/3    00:00:00 /home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin/xpcshell -g /home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin -v 170 -f /home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin/components/httpd.js -e const _PROFILE_PATH = '/tmp/tmpOpIykV.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = true; -f /home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/_tests/testing/mochitest/server.js
SUITE-START | Running 1 tests
dir: editor/libeditor/tests
pk12util: PKCS12 IMPORT SUCCESSFUL
MochitestServer : launching [u'/home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin/xpcshell', '-g', u'/home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin', '-v', '170', '-f', u'/home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpaSbEDp.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '127.0.0.1'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = true;", '-f', '/home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/_tests/testing/mochitest/server.js']
runtests.py | Server pid: 22470
runtests.py | Websocket server pid: 22473
runtests.py | SSL tunnel pid: 22477
failed to bind socket on port 4443: error -5982
Shutting down...
[22470] WARNING: failed to bind socket: file /home/ehsan/moz/src.try/netwerk/base/nsServerSocket.cpp, line 361

!!! could not start server on port 8888: [Exception... "Component returned failure code: 0x804b0036 (NS_ERROR_SOCKET_ADDRESS_IN_USE) [nsIServerSocket.init]"  nsresult: "0x804b0036 (NS_ERROR_SOCKET_ADDRESS_IN_USE)"  location: "JS frame :: /home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin/components/httpd.js :: _start :: line 550"  data: no]
$ ps aux | grep ssltunnel
ehsan    13937  0.0  0.0  32324 11164 pts/3    Sl   16:45   0:00 /home/ehsan/moz/src.try/obj-ff-clang-plugin.noindex/dist/bin/ssltunnel /tmp/ssltunnelWyz8Po.cfg


What has happened here is I have an ssltunnel process lying around from a previous run of mochitest.  I always kill this manually and retry, and I have literally done this tens of thousands of times so far.  It would really be nice if our tools assisted us here by figuring out if the port is held open by such a process we have opened before, and kill it if so and retry automatically.

Comment 1

[Geoff Brown [:gbrown]]

See https://dxr.mozilla.org/mozilla-central/rev/ef9a0f01e4f68214f0ff8f4631783b8a0e075a82/testing/mochitest/runtests.py#1641.

I believe the issue here is that the ppid is not 1, so it is not considered an orphan. As I recall, I put that check in because I was feeling uneasy about killing an ssltunnel (or xpcshell or whatever) that might be running for another job (running 2 tests in parallel?).

Comment 2

[(no longer active)]

How can one run two tests in parallel?  The second one is going to launch an ssltunnel which is going to try to bind to the port that the first one is listening to.

Comment 3

[Geoff Brown [:gbrown]]

Good point!

So, better to kill any ssltunnel or xpcshell found running at the start of the test run? I'll try that out...

Comment 4

[Geoff Brown [:gbrown]]

Attachment: kill any xpcshell or ssltunnel already running at start of mochitest or remote reftest

This works well for me locally and does not seem to cause any trouble in automation:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=ed03b5a8c0cd6fe4d8d7597e6407e0974bc28a0e

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8902323 [details] [diff] [review]
kill any xpcshell or ssltunnel already running at start of mochitest or remote reftest

Review of attachment 8902323 [details] [diff] [review]:
-----------------------------------------------------------------

good stuff

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b823779242ed
Kill any xpcshell or ssltunnel already running at start of tests; r=jmaher

Comment 7

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/b823779242ed

Comment 8

[(no longer active)]

Thank you so much!  If I say I may have spent days of my life overall hunting down these rouge ssltunnel processes and killing them manually, I don't think I'm overstating it.  :-)