1152994 - Improve mochitest orphan process checks

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

In bug 1084614, I tried to resolve the problem of ssltunnel and xpcshell processes left over from previous test runs causing infrequent, intermittent test failures. The strategy was to kill orphan ssltunnel and xpcshell processes and log all such running processes found at the start of a test run.

That patch has helped, significantly on Android.

But bug 915212 (and maybe bug 919920) shows that there are additional cases to be considered.

1. B2G emulator mochitests do not use the code introduced in bug 1084614.

2. On Ubuntu VM 12.04 (only?) we still sometimes find ssltunnel and xpcshell running at the start of the test, started recently (30 to 60 minutes earlier, typically), but not orphaned. Presumably the parent is runtests.py, but we don't know for sure. We do not kill non-orphaned processes to account for Android foopies, where we expect multiple ssltunnel and xpcshell instances to be running concurrently.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: add orphan server check to B2G mochitest

Simple patch to handle the B2G case.

Comment 2

[Geoff Brown [:gbrown]]

I haven't figured out what to do about case #2 yet.

Comment 3

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/4ae5f2a4a19d

Comment 4

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/4ae5f2a4a19d

Comment 5

[Geoff Brown [:gbrown]]

I wanted to keep this open to consider adding more diagnostics. With recent progress in bug 915212, I think we probably have enough information and understanding of root cause (for example, if the network connection is lost, buildbot will abandon the test and recycle the instance, but there may not be a reboot, so the old test may still be running when a new one is starting).