625060 - reftest-ipc hanging on shutdown during crashtest run and during startup on reftest run

Status: RESOLVED FIXED

(Core :: IPC, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Crashtest problem:

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1294822125.1294823107.13624.gz
Rev3 MacOSX Leopard 10.5.8 tryserver debug test crashtest on 2011/01/12 00:48:45 
s: talos-r3-leopard-021
TEST-UNEXPECTED-FAIL | Shutdown | application timed out after 330 seconds with no output
TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks!
TEST-UNEXPECTED-FAIL | tab process 302 | automationutils.processLeakLog() | missing output line for total leaks!

Log doesn't shed any light, although

WARNING: nsAppShell::Exit() called redundantly: file /builds/slave/try-osx-dbg/build/widget/src/cocoa/nsAppShell.mm, line 762

seems to imply we're at least /trying/ to shut down.


Reftest problem:

http://tinderbox.mozilla.org/showlog.cgi?log=MozillaTry/1294822148.1294822827.12582.gz
Rev3 MacOSX Leopard 10.5.8 tryserver debug test reftest on 2011/01/12 00:49:08 
s: talos-r3-leopard-046
TEST-UNEXPECTED-FAIL | automation.py | application timed out after 330 seconds with no output
TEST-UNEXPECTED-FAIL | automationutils.processLeakLog() | missing output line for total leaks!
TEST-UNEXPECTED-FAIL | tab process 295 | automationutils.processLeakLog() | missing output line for total leaks!

Nothing in the spew looks particularly telling.  I see

nsNativeModuleLoader::LoadModule("/Users/cltbld/talos-slave/test/build/MinefieldDebug.app/Contents/MacOS/components/libxpcomsample.dylib") - load FAILED, rv: 80004005, error:
	Unknown error: -2804

but I presume we can cope with that.

Comment 1

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

See bug 625058 comment 1, which was supposed to be posted here.

I fixed up the IPDL tests with a voodoo patch (GRE dir problem), and they all passed.  Yay?  So, likely a problem with the TYPE_MOZILLA_CHILD MessageLoop on mac, since the normal TYPE_UI loop is used in plugins and IPDL tests which don't seem to have these problems.

Comment 2

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

I can reproduce this problem in test-ipc.xul.  The content process doesn't respond to messages from the parent for a long time, then occasionally gets dislodged after the browser sends several messages.

Comment 3

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

If we're serious about fennec extension developers using mac nightlies, then this ought to block for similar reasons as bug 617860 did.  The bug I'm about to post a patch for is quite bad.

Comment 4

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: Only plugin-container needs to walk out of plugin-container.app/Content/MacOS to find the real GRE dir

Kinda sneaking this in here.  My first debugging tack was to run the ipdl unit tests, which don't work without this patch.

Comment 5

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: Process tasks that might otherwise have been ignored because of failed NS_DispatchToMainThread() early in startup

Tasks being ignored during startup led to a nasty problem in RPCChannel: since RPCChannel keeps its own message queue, mPending, then when the subprocess received Messages *after* startup stabilized, it would process one of the old, previously-ignored messages from the queue.  The Q would then forever be K messages behind. (Or until an RPC was sent, which shouldn't happen in normal operation.)  This was incredibly annoying in the test browser, and prevented reftests from working.

Comment 6

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Attachment: Make sure XRE_ShutdownChildProcess() actually does that on mac

Comment 7

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

This should also fix bug 575918, if bug 575918 comment 9 is correct wrt test slaves.

Comment 8

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

No dice.  To the back burner again.

Comment 9

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Stuart, I'm interpreting this blocking- as meaning that fennec-on-mac bugs aren't worth fixing, so I'm not going to fix any more of them.  Please make sure everyone agrees with that.

Comment 11

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Requesting re-triage because there was apparently a misunderstanding about the severity of this bug.  See bug 602539 for other symptoms.

Comment 12

[Mark Finkle (:mfinkle) (use needinfo?)]

blocking for Mac IPC failure

Comment 13

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Ben, any ETA on reviews?

Comment 14

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

http://hg.mozilla.org/mozilla-central/rev/6ad50f1f840f
http://hg.mozilla.org/mozilla-central/rev/4ef52d80176f
http://hg.mozilla.org/mozilla-central/rev/f4199091704c