605313 - nsBaseAppShell generates busy-wait event storm on main thread at shutdown; causing orange xpcshell tests

Status: RESOLVED DUPLICATE of bug 605314

(Core :: Widget, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

nsBaseAppShell registers itself as a thread observer to insert the native event loop.  It injects dummy (mozilla, non-native) events to effectively invert the loop cycle so that it can use the native event loop in a (potentially) blocking fashion.

At shutdown it forbids the native event loop from being blocking:
    if (mExiting)
      mayWait = PR_FALSE;

    mLastNativeEventTime = PR_IntervalNow();
    if (!DoProcessNextNativeEvent(mayWait) || !mayWait)
      break;

It's pretty straightforward to see how the always-active dummy insertion logic defeating nsThread's blocking mode combined with disabling the native event loop's ability to block would cause busy-waiting.  If you attach a debugger to follow this, be forewarned that the generation of the dummy event triggers nsBaseAppShell::OnDispatchedEvent which is not suppressed/ignored so nsAppShell::ScheduleNativeEventCallback gets called and then a bunch of additional control flow happens to deal with that irrelevant native event.

I believe this is leading to intermittent oranges because the busy-wait on the main thread fights all of the other threads that are attempting to shutdown.  Given enough time the other threads will eventually close out their own event queues and respond with shutdown acks that allow XPCOM to shutdown.  The problem is that the buildbot may kill the test before that happens.  The longer the backlog of events on other threads, the worse the problem can be.  In some of our tests, the test completes with a significant number of outstanding events on a mozStorage asynchronous thread.

I make the modest proposal that the logic be modified to always let someone block.  So, at mExiting time, we stop generating dummy events.  With the provided patch I no longer see any dummy events injected during my xpcshell run during shutdown.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

This appears to have been stuck in the maw of bugzilla for several months; just got the initial e-mail for this bug, which is what I submitted for bug 605314.  Duping to that.