340022 - Virtual machine deadlock can happen on finalization of Java XPCOM proxies

Status: RESOLVED FIXED

(Core Graveyard :: Java to XPCOM Bridge, defect)

Description

[Tudor TEUSAN]

User-Agent:       Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.3) Gecko/20060426 Firefox/1.5.0.3
Build Identifier: 

CONTEXT :

The java garbage collector can be (and in many actual implementations is) multi-threaded. A multi-threaded gc often uses a special, finalizer thread for object finalization.

Suppose we have java app with 2 threads: the main one - MainThread - and the finalizer one - FinalizeThread. 
MainThread is the UI thread for both the app and the embedded mozilla and works around an event loop. Our app also contains (at least) one displayed mozilla window. An event happens in the mozilla window (like a mouse click). It is read in the event loop and dipatched to mozilla. 
The general  pattern of event processing in mozilla (see PL_ProcessPendingEvents()) is : 
a. Enter monitor of the thread
b. Process events
c. Exit monitor of the thread

Step b. (that actually happens in http://lxr.mozilla.org/mozilla1.8.0/source/xpcom/threads/plevent.c#1327) can translate to a call to java code (e.g. a click in the mozilla window can generate a SWT.Activate event that will be dispatched to various registered event listeners). Let's suppose that this java code calls (through jni) an xpcom method. 

While MainThread is processing step b, FinalizerThread starts running and finalizing a proxy (is in finalizeProxy() in nsJavaWrapper.cpp)

BUG :

1. FinalizerThread acquires the global gJavaXPCOMLock (http://lxr.mozilla.org/mozilla1.8.0/source/extensions/java/xpcom/nsJavaWrapper.cpp#1731).
2. FinalizerThread deletes the corresponding JavaXPCOMInstance (http://lxr.mozilla.org/mozilla1.8.0/source/extensions/java/xpcom/nsJavaWrapper.cpp#1754).
  The destructor of JavaXPCOMInstance gets invoked (http://lxr.mozilla.org/mozilla1.8.0/source/extensions/java/xpcom/nsJavaXPCOMBindingUtils.cpp#688).
  As we are not on the main thread the NS_ProxyRelease() calls in the above mentioned destructor will need to acquire the xpcom monitor of the MainThread. MainThread currently own its monitor so FinalizerThread BLOCKS waiting for MainThread

3. MainThread proceeds with the xpcom call. As it uses or returns a xpcom object (this is one last supposition) it will call the NativeToJavaProxy::Find() method. This method starts by trying to acquire the global gJavaXPCOMLock which is held by FinalizeThread. So MainThread BLOCKS waiting for FinalizeThread.

4. DEADLOCK.

This seems a highly hypothetical scenario but it actually happens. With the hack described in bug 339947 it is pretty frequent.

Reproducible: Sometimes




Proposed solution :

This is a simple one. In the finalizeProxy() method (nsJavaWrapper.cpp) we should explicitly release the gJavaXPCOMLock autolock (through lock.unlock()) before deleting the JavaXPCOMInstance :

 if (gJavaXPCOMLock) {
    nsAutoLock lock(gJavaXPCOMLock);
    ...
    if (gJavaXPCOMInitialized) {
      // Get native XPCOM instance
      nsresult rv = GetXPCOMInstFromProxy(env, aJavaProxy, &xpcom_obj);
      if (NS_SUCCEEDED(rv)) {
        JavaXPCOMInstance* inst = NS_STATIC_CAST(JavaXPCOMInstance*, xpcom_obj);
	...
        // !!!!  UNLOCK BEFORE DELETING
        lock.unlock(); 
        // !!!!  THIS IS ALL
	delete inst;
      }

Comment 1

[Tudor TEUSAN]

Attachment: patch

Patch for the above mentioned solution.

Comment 2

[jhp (no longer active)]

Checked in to trunk and 1.8 branch.  ->FIXED